Active Transport Based Notifications

ABSTRACT

Techniques for triggering an action associated with a task items are provided. A task item associated with a triggering criterion is provided. The triggering criterion requires an occurrence of any of a plurality of communication events. The plurality of communication events includes a telephone call. In some implementations, at least two of the plurality of communication events are of different communication types. In some implementations, the different types are telephone calls, emails, and text messages. The occurrence of a first communication event of the plurality of communication events is detected. It is determined the triggering criterion is satisfied. In response to the determining, a notification associated with the task item is caused to be presented to a user of the electronic device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation-in-Part of U.S. patentapplication Ser. No. 13/251,118, filed Sep. 30, 2011, entitled“Performing Actions Associated with Task Items that Represent Tasks toPerform,” which in turn claims priority to U.S. Provisional PatentApplication No. 61/493,201, filed Jun. 3, 2011, entitled “Generating andProcessing Data Items that Represent Tasks to Perform,” both of whichare incorporated by reference for all purposes as if fully set forthherein.

This application is related to U.S. patent application Ser. No.12/479,477, filed Jun. 5, 2009, entitled “Contextual Voice Commands,”the entire contents of which are hereby incorporated by reference as iffully set forth herein.

This application is related to U.S. patent application Ser. No.12/987,982, filed Jan. 10, 2011, entitled “Intelligent AutomatedAssistant,” the entire contents of which are hereby incorporated byreference as if fully set forth herein.

This application is related to U.S. patent application Ser. No.13/251,088, filed Sep. 30, 2011, entitled “Generating and ProcessingTask Items that Represent Tasks to Perform,” the entire contents ofwhich are hereby incorporated by reference as if fully set forth herein.

This application is related to U.S. patent application Ser. No.13/251,104, filed Sep. 30, 2011, entitled “Triggering NotificationsAssociated with Tasks Items that Represent Task to Perform,” the entirecontents of which are hereby incorporated by reference as if fully setforth herein.

This application is related to U.S. patent application Ser. No.13/251,127, filed Sep. 30, 2011, entitled “Organizing Task Items thatRepresent Tasks to Perform,” the entire contents of which are herebyincorporated by reference as if fully set forth herein.

FIELD OF THE INVENTION

The present invention relates to electronic reminders and, moreparticularly to, the intelligent generation, organization, triggering,and delivery of reminders and tasks in electronic to-do lists.

SUMMARY

In accordance with some implementations, a method for triggering anaction associated with a task is disclosed. In some implementations, themethod is performed at an electronic device having one or moreprocessors and memory storing one or more programs for execution by theone or more processors. The method includes receiving a task itemassociated with a triggering criterion, wherein the triggering criterionrequires an occurrence of any of a plurality of communication events,wherein the plurality of communication events includes a telephone call.The method further includes detecting the occurrence of a firstcommunication event of the plurality of communication events. The methodfurther includes determining that the triggering criterion is satisfied.The method further includes, upon determining that the triggeringcriterion is satisfied, causing a notification associated with the taskitem to be presented to a user of the electronic device.

In some implementations, the first communication event is a telephonecall. In some implementations, at least two of the plurality ofcommunication events are of different communication types. In someimplementations, the different communication types include a telephonecall and an email. In some implementations, the different communicationtypes include a telephone call and a text message. In someimplementations, the different communication types include a telephonecall and a video chat.

In some implementations, the first communication event includes content,and the triggering criterion requires that the content relate to apredefined subject. In some such implementations, the method furthercomprises determining that the triggering criterion is satisfied basedat least in part on a determination that the content relates to thepredefined subject. In some implementations, the content is text from anemail or a text message.

In some implementations, the first communication event is a telephonecall, and the method further comprises obtaining a text stringcorresponding to one or more utterances spoken during the telephone callby at least one participant in the telephone call, wherein thedetermination that the content relates to the predefined subjectincludes determining that one or more words in the text string relate tothe predefined subject.

In some implementations, the triggering criterion further requires thatthe communication event be a communication with a specified person.

In some implementations, the first communication event is a textmessage, and causing the notification to be presented comprises causingan affordance relating to the first task item to be displayed on theelectronic device near a representation of the text message. In someimplementations, the affordance is a button that, when selected by theuser, causes an action associated with the task item to be performed.

In some implementations, the first communication event is an email, andcausing the notification to be presented comprises causing an affordancerelating to the first task item to be displayed on the electronic devicenear a representation of the email. In some implementations, theaffordance is a button that, when selected by the user, causes an actionassociated with the task item to be performed.

In some implementations, the first communication event is a telephonecall, and causing the notification to be presented comprises causing anaudio output relating to the first task item to be produced by theelectronic device. In some implementations, the electronic device is atelephone, and the audio output is produced while the telephone isringing. In some implementations, the audio output is produced during atelephone conversation between a user of the electronic device and aparticipant of the telephone call. In some implementations, the audiooutput is produced at a lower volume than a voice of the participant ofthe telephone call.

In accordance with some implementations, another method for triggeringan action associated with a task is disclosed. In some implementations,the method is performed at an electronic device having one or moreprocessors and memory storing one or more programs for execution by theone or more processors. The method includes providing a task itemassociated with a triggering criterion, wherein the triggering criterionrequires an occurrence of any of a plurality of communication events,wherein at least two of the plurality of communication events are ofdifferent communication types. The method further includes detecting theoccurrence of a first communication event of the plurality ofcommunication events. The method further includes determining that thetriggering criterion is satisfied. The method further includes, upondetermining that the triggering criterion is satisfied, causing anotification associated with the task item to be presented to a user ofthe electronic device.

In some implementations, the triggering criterion requires that thefirst communication event be associated with a specified person. In someimplementations, the first communication event is associated with thespecified person when the specified person initiates the firstcommunication event or receives the first communication event from theuser.

In accordance with some implementations, yet another method fortriggering an action associated with a task is disclosed. In someimplementations, the method is performed at an electronic device havingone or more processors and memory storing one or more programs forexecution by the one or more processors. The method includes providing atask item associated with a triggering criterion, wherein the triggeringcriterion requires an occurrence of a telephone call at an electronicdevice. The method further includes detecting the occurrence of thetelephone call. The method further includes, in response to detectingthe occurrence of the telephone call, causing a notification associatedwith the task item to be presented to a user of the electronic device,wherein the notification is separate from a telephone call notification.

In accordance with some implementations, a computer-readable storagemedium (e.g., a non-transitory computer readable storage medium) isprovided, the computer-readable storage medium storing one or moreprograms for execution by one or more processors of an electronicdevice, the one or more programs including instructions for performingany of the methods described herein.

In accordance with some implementations, an electronic device (e.g., aportable electronic device) is provided that comprises means forperforming any of the methods described herein.

In accordance with some implementations, an electronic device (e.g., aportable electronic device) is provided that comprises a processing unitconfigured to perform any of the methods described herein.

In accordance with some implementations, an electronic device (e.g., aportable electronic device) is provided that comprises one or moreprocessors and memory storing one or more programs for execution by theone or more processors, the one or more programs including instructionsfor performing any of the methods described herein.

In accordance with some implementations, an information processingapparatus for use in an electronic device is provided, the informationprocessing apparatus comprising means for performing any of the methodsdescribed herein.

BACKGROUND

People have devised numerous ways to remind themselves of certain tasksor events. Many people have and still use physical media to write on,such as sticky notes and calendars. With the ubiquity of electronicdevices, many people have turned to computers to help manage their to-dolists and keep a record of upcoming events. Numerous reminder and to-doapplications are available, both for desktop computers as well ashandheld devices, such as laptop computers, tablet computers, and“smart” phones.

However, the timeliness and accuracy of a notification provided to auser of a reminder application depends almost entirely on input receivedfrom the user. For example, if a user enters a reminder with the wrongdate for an important event, then the user might not receive anotification of the event until after the event has occurred. As anotherexample, if a user provides a generic description of a task (e.g., “sendhim an email”) in a to-do application, then, when the user later readsthe description, the user might not remember who “him” is and/or whatthe content of the email should be. In other words, when it comes toreminder and to-do applications, the old adage of “garbage in garbageout” is applicable.

Moreover, users do not always have the time or willingness to set atriggering condition for each and every event that should trigger areminder notification, and reminder and to-do applications may not beable to monitor each type of event or condition that may be a usefultrigger. For example, for a reminder such as “remind me to call my momafter I hear from my sister,” traditional reminder and to-doapplications would not monitor different communications types so that areminder notification could be issued regardless of how the user “heardfrom” her sister.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram that depicts a system that is used forcreating and processing task items, according to an embodiment of theinvention;

FIG. 2 is a flow diagram that depicts a process for generating a taskitem based on context of user input, according to an embodiment of theinvention;

FIG. 3 is a flow diagram that depicts a process for determining a timeto provide a reminder to a user ahead of a scheduled time for a task,according to an embodiment of the invention;

FIG. 4 is a view of a travel reminder, according to an embodiment of theinvention;

FIGS. 5A-15 depict views of various types of lists, according to anembodiment of the invention;

FIG. 16 is a block diagram that illustrates a computer system upon whichan embodiment of the invention may be implemented; and

FIG. 17 is a flow diagram that depicts a process for initiating actionsassociated with task items, according to an embodiment of the invention.

FIGS. 18A-20 are flow diagrams depicting methods for triggering actionsassociated with tasks, according embodiments of the invention.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Multiple techniques are provided below for assisting a user in managinghis/her tasks. Tasks are represented as task items in a computingsystem. Specifically, approaches for generating task items, organizingtask items, triggering the notification of tasks based on correspondingtask items, and consuming task items are described in detail below.

With respect to generating a task item, a task item is generated basedon input from a user. The input (whether voice or text) includes one ormore details about a task while context of the input is used todetermine one or more other details about the task. The one or moreother details are not reflected in the input. Context may include datathat is stored on a device of the user. For example, a user provides thefollowing voice input: “Call George at 5 PM today.” The user may have acontact list that includes information about multiple people with thefirst name of George. However, based on the context of the input, suchas a recent voice message from a phone number associated with aparticular George, it can be determined which George the user intends tocall. In this way, a user is not required to be specific about eachdetail of a task when providing input about the task.

With respect to triggering a notification of a task item, one or morecharacteristics of a device may be used to determine whether to triggerthe notification. Thus, time is not the sole factor (if at all) ofwhether a notification should be provided to a user. Examples of suchcharacteristics may include where the device is located, what the deviceis displaying or processing, and specific movements of the device. Forexample, the fact that the device is in a car or at work may trigger thegeneration of a reminder of a task. As another example, if the device iscurrently displaying web content, then a user of the device may beconsidered to be “online,” which status might trigger a notification ofa task to be provided to the user. As yet another example, communicationevents, such as emails, telephone calls, text messages, and the like,can trigger a notification of a task to be provided to the user.

With respect to “consuming” task items, instead of simply providing areminder of a task, the task is automated so that a user is not requiredto perform the steps typically required to perform the task. Forexample, a user may want to call a particular person at a particulartime. When the particular time equals the current time, instead ofsimply reminding the user about the call, the call can be set up, readyto make without the user having to specify the person's phone number.

With respect to organizing task items, task items may be organizedautomatically or manually into multiple lists. Each list corresponds toa different attribute of a task item, such as the type of task, the typeof criteria that is used to trigger a notification of a task, and thelocation of where the task is to be performed.

Task Items

Again, a “task item” is an electronic data item that contains one ormore details about a task to perform, whether by a user or automaticallyby a process. A task item is generated based on input from a user. Atask item may be one of two types: tasks associated with reminders(“reminder task”) and tasks not associated with reminders (“non-remindertask”). A reminder task is associated with an action, such as anotification being generated and provided to a user, while anon-reminder task is not associated with any action. A non-reminder taskmay be associated with a “complete-by” date or time. However, thecomplete-by date or time does not trigger the creation of a notificationor other action. In other words, while a reminder task is associatedwith one or more triggering criteria that, when satisfied, trigger anaction, a non-reminder task is not. Thus, a “task item” may or may notbe associated with one or more triggering criteria that, when satisfied,triggers an action.

System Overview

FIG. 1 is a block diagram that depicts a system 100 that is used forcreating task items and processing task items, according to anembodiment of the invention. System 100 includes a device 110, a network120, and a cloud 130.

Device 110 is any computing device that is capable of receiving inputfrom a user and displaying information about tasks. Non-limitingexamples of device 110 include a desktop computer and a handheld device,such as a laptop computer, a tablet computer, and a “smart” phone. Inthe illustrated embodiment, device 110 includes a task manager 112. Taskmanager 112 processes task items, both of the reminder type or of thenon-reminder type. Task manager 112 may be implemented in software,hardware, or any combination of software and hardware.

Device 110 includes communication technology (e.g., wireless technology)for sharing information with other devices. Device 110 can include avariety of integrated user interface units or can be coupled to userinterface units through one or more communication ports or data links ofthe device. Non-limiting examples of user interface units include avoice input unit (e.g., a microphone), physical input units (e.g., akeyboard, a mouse, a track ball, a rotary dial or wheel, a touchpad, ora touch-screen), and motion sensors (e.g., an accelerometer,magnetometer, or a gyroscope). Any of these user interface units can beimplemented as an external unit that communicates with device 110 toprovide user input using a wired or wireless communication technology.Examples of wired communication technology include Universal Serial Bus(USB) interface, FireWire interface, etc. Examples of wirelesscommunication technology include Bluetooth, Wi-Fi, and WiMax, infrared.Through these user interface units, device 110 can receive physical orvoice inputs from the user.

Device 110 includes one or more output units to present visual and audioinformation to a user. Non-limiting examples of output units include adisplay unit for displaying visual data and a speaker for playing audio.

Cloud 130 is implemented by one or more computing devices. Cloud 130hosts multiple services, such as a NLP (natural language processing)service 132 and one or more other services 134A-N. NLP service 132 usesone or more models of real-world things that a user can talk about inorder to make sense of what the user is trying to say. For example, NLPservice 132 can determine, based on models and context, what a user maybe referring to when the user uses terms like, “him,” “there,” or“that.” An example of how NLP service 132 might operate is described inU.S. patent application Ser. No. 12/987,982, referenced above.

NLP service 132 may employ numerous APIs to communicate with andinitiate actions performed by the one or more other services 134A-N and,optionally, other services not hosted in cloud 130. For example, inresponse to voice data sent from device 110, where the voice datareflects the user command “Reserve two seats at Maribella's in San Joseat 7 PM tonight,” NLP service 132 makes an API call to an onlinereservation service provided by Maribella's restaurant to initiate thecreation of two reservations at that restaurant for 7 PM. Thus, NLPservice 132 allows many operations to be performed automatically withoutrequiring a user of device 110 to manually input text data and interactwith numerous applications.

Communication between device 110 and services hosted in cloud 130 ismade possible via network 120. Network 120 may be implemented by anymedium or mechanism that provides for the exchange of data betweenvarious computing devices. Examples of such a network include, withoutlimitation, a network such as a Local Area Network (LAN), Wide AreaNetwork (WAN), Ethernet or the Internet, or one or more terrestrial,satellite, or wireless links. The network may include a combination ofnetworks such as those described. Without limitation, the network maytransmit data according to Transmission Control Protocol (TCP), UserDatagram Protocol (UDP), and/or Internet Protocol (IP).

The following description includes numerous examples where both device110 and cloud 130 take part in generating task items, organizing taskitems, triggering notifications of task items, and consuming task items,and examples where such operations are performed by either the device110 or the cloud 130 alone. Instead, one or more of the techniquesdescribed herein may be implemented wholly on device 110 (making network120 and cloud 130 unnecessary), wholly in cloud 130, or using anycombination of device 110 and cloud 130.

Processing of Task Items

Task items may be created on device 110 or in cloud 130 based on inputreceived at device 110. Although not depicted, task items may be storedon device 110 or in cloud 130, or synchronized to both. If task itemsare stored in cloud 130, then task manager 112 may retrieve the taskitems in response to, for example, input from a user or the one or moretriggering criteria associated with one or more task items beingsatisfied.

In the scenario where task items are created and stored in cloud 130,task manager 112 may be, primarily, a set of one or more user interfacesthat display information about tasks. Thus, a task service (not shown)in cloud 130 would be responsible for maintaining task items andtriggering any notifications when triggering events occur.

Alternatively, task manager 112 creates and stores task items on device110. In this scenario, task manager 112 may be entirely responsible formaintaining task items and generating any notifications when triggeringevents occur. One advantage of this scenario is that device 110 may beoperating in an “offline” mode where device 110 is not capable ofcommunicating with any service hosted in cloud 130.

Further, in this scenario, device 110 may include a service like NLPservice 132, which may be part of task manager 112 or may executeseparately from task manager 112. Such a service acts as aconversational interface to allow a user to quickly and easily createtasks. Such a service may be implemented by a process that iscontinually executing in the background without requiring a user ofdevice 110 to provide input to cause the service to execute. Thus,whenever device 110 starts up (or restarts), the service isautomatically started.

Alternatively, information needed to create task items may be identifiedby NLP service 132 (i.e., hosted in cloud 130). Device 110 may include auser input interface that continuously executes in the background,identifies input (e.g., voice or text) from a user, and sends the inputover network 120 to NLP service 132. Once NLP service 132 identifiestask details in the input, NLP service 132 may send task information(e.g., a description of a task and a time to complete the task) (a) overnetwork 120 to task manager 112, which creates and stores a task itembased on the information or (b) to a task service in cloud 130 to createa task item based on the information.

Most of the examples provided herein involve NLP service 132 receivinginput data from device 110, identifying details (about a task) reflectedin the input data, and providing those details to task manager 112.However, embodiments of the invention are not limited to this scenario.Such examples may alternatively involve only device 110 or may involvedevice 110 as merely an input and display device where NLP service 132and a task service in cloud 130 provide the primary functionality.

I. Generating Task Items Based on Context

According to an embodiment of the invention, a task item is generatedbased on input and context of the input. “Context” of input refers todata that is currently or recently (relative to input, from a user, thatinitiated the generation of a task item) displayed or processed atdevice 110. Thus, context data is not reflected in the input from theuser. For example, a user of device 110 may provide the following voiceinput: “Send him an email about the project when I get home.” Thepronoun “him” is ambiguous because it is not clear, from the inputalone, to whom “him” refers. However, the context of the voice input maybe that device 110 currently displays (or just recently displayed) anemail from an individual named Jack Bauer where the email includes arequest for a status update about a project named “Bunny.” Based on thevoice input and the context, task manager 112 (or a task service incloud 130) creates a task item that includes the description “Send JackBauer an email about Project Bunny” and that includes the triggeringcriterion of device 110 being at a geographical location that is at ornear the user's home. When device 110 is at or near the user's home,task manager 112 causes a notification to be displayed on device 110where the notification includes the description from the task item.

FIG. 2 is a flow diagram that depicts a process 200 for generating atask item based on context of user input, according to an embodiment ofthe invention. At step 210, input that expressly specifies one or morefirst attributes for a task is received from a user. The input may betext input or voice input. The text input may be from a user of device110 pressing physical keys on device 110 or pressing a touch screen ofdevice 110 that includes a graphical keyboard. Additionally oralternatively, device 110 includes a microphone that accepts, from auser, voice input that device 110 converts into voice data. Device 110may send the input data (whether voice data or text data) to NLP service132, which analyzes the input data to identify the one or more firstattributes for the task. Instead, as noted previously, device 110 mayinclude functionality to analyze the input data to identify the one ormore first attributes for the task. (Although many of the examplesherein refer to natural language processing, natural language processingis not required.)

At step 220, a task item is generated for the task based on the inputdata. At step 230, one or more second attributes for the task aredetermined based on context data that is separate from the input.Although step 230 is depicted as occurring after step 220, step 230 mayoccur before step 220.

At step 240, the one or more first attributes and the one or more secondattributes are stored in association with the task item.

The steps of process 200 may be performed by one or multiple devices.For example, the input in step 210 may be processed at device 110 togenerate the task item. In this scenario, task manager 112 (or anotherprocess executing on device 110) identifies the context data associatedwith the input to determine the one or more second attributes, for thetask, that are not identified in the input. Task manager 112 then storesthe one or more second attributes in or in association with the taskitem.

Alternatively, in another scenario, device 110 sends the user input overnetwork 120 to NLP service 132. NLP service 132 accepts, as input,context data associated with the input to determine the one or moresecond attributes, for the task, that are not identified in the input.Context data may have been sent to NLP service 132 prior to the inputthat initiates the generation of the task item (in step 220). NLPservice 132 sends the one or more second attributes to task manager 112(or a task service in cloud 130). Task manager 112 stores the one ormore second attributes in or in association with a newly-generated taskitem.

Certain words or phrases may be used to cue NLP service 132 tocommunicate with manager 112. For example, user commands that begin with“Remind me . . . ” and “I need to. . . ” are used by NLP service 132 todetermine to communicate with task manager 112. In response to detectingone of those user commands, NLP service 132 analyzes the input data(from device 110) and, optionally, context data for certain types oftask details, such as a location, time, description, and/or action. NLPservice 132 then determines to communicate with task manager 112 andsends, to task manager 112, the task details as part of thecommunication(s).

Sources of Context Data

Context data associated with user input that initiates the generation ofa task item may come from one of many sources. Non-limiting examples ofcontext data include data that is or was displayed on device 110(“display data”), data that is stored on or in association with device110 (“personalized data”), data that is or was processed by device 110(“process data”), data that was previously provided by a user of device110 (“input data”), data that indicates the location of device 110(“location data”).

The following is an example of display data, or data that is or wasdisplayed on device 110. Device 110 displays a map that includes amarker associated with a specific location on the map. A user of device110 then says, while the map is displayed or soon after the map wasdisplayed, “I need to be there by 5 today.” NLP service 132 (or a voiceanalyzer on device 110) analyzes voice data that reflects the voiceinput. NLP service 132 analyzes data that is currently displayed ondevice 110 to determine what “there” refers to. NLP service 132identifies the marker and the associated location and replaces “there”with the location. NLP service 132 sends, to task manager 112, task datathat indicates 5 PM today as the completion time of the task and thespecified location as the location of the task. Task manager 112generates a task item based on the task data.

As another example of display data, device 110 displays an email that isfrom a particular sender and includes a subject line. A user of device110 then says, “I need to email him about that subject in two hours.”Device 110 sends voice data that reflects this input and an image ofwhat is displayed to NLP service 132. In response, NLP service 132identifies the email address of the sender of the email and the subjectof the email. NLP service 132 sends, to task manager 112, task data thatindicates a time of two hours from the current time as the completiontime of the task and

The following is an example of personalized data, or data that is storedon or in association with device 110. A user of device 110 says, “I willhave lunch with Rachelle tomorrow at 12 noon.” Device 110 sends voicedata that reflects this input to NLP service 132, which identifies“Rachelle” in the voice data. NLP service 132 looks up “Rachelle” incontact data or an “address book” (stored on device 110 or in cloud 130)and determines that the last name of Rachelle is Goodwin. NLP service132 then causes “Rachelle Goodwin” to be associated with a task itemthat is generated for the task. In addition to or instead of beingstored on device 110, personalized data may be stored in cloud 130,i.e., remote to device 110.

The following is an example of process data, or data that was recentlyprocessed by device 110. For example, a user of device 110 used device110 as a phone to communicate with a friend. Device 110 keeps track ofwho the user recently spoke with. After ending the call, the user says,“Remind me to call her back in 30 minutes.” NLP service 132, in additionto analyzing the voice input, analyzes data that indicates who recentlyestablished a phone call with device 110 (e.g., the last five phonecalls). NLP service 132 determines the phone number of the most recentlyestablished phone call with device 110. NLP service 132 then determines,based on contact data, that the phone number is associated withparticular individual. NLP service 132 sends, to task manager 112, taskdata that indicates a task of calling, a time of 30 minutes from thecurrent time as the completion time of the task, the name of theparticular individual, and, optionally, the phone number of theparticular individual. Task manager 112 generates a task item based onthe task item.

The following is an example of input data, or data that was recently(e.g., the last 5 minutes) provided by a user of device 110. The inputfrom the user may be text input or voice input. Device 110 or NLPservice 132 keeps track of recently entered input and may use that inputto determine the identity of certain terms reflected in current input.For example, a user of device 110 says, “Remind me to meet him there at7 tonight.” NLP service 132 receives voice data that reflects that voiceinput and identifies the terms “him” and “there.” Although it is notclear who “him” is and where “there” is, NLP service 132 accesses inputthat was recently received from the user. Such recently-received inputreflects the names “George Reed” (identified as a name of a person) and“Starbucks” (identified as a place). In response, NLP service 132 causesa task item to be generated where the task is to “Meet George Reed atStarbucks” where the time is 7 PM of the current day.

The following is example of location data, or data that indicates alocation of device 110, whether current or past. A user of device 110says, “Remind me to meet Julian here next Thursday for lunch.” Device110 sends voice data that reflects this input to NLP service 132. NLPservice 132 identifies the term “here” and, in response, determineswhere device 110 is currently located. The current location may bedetermined in numerous ways. For example, device 110 may provide, to NLPservice 132, a geographical location, such as longitude and latitudecoordinates. NLP service 132 may then determine, based on thecoordinates, a name of the place or establishment that is located atthose coordinates. NLP service 132 causes a name of the place orestablishment to be associated with a task item for the task to meetJulian for lunch on the date indicated.

Alternatively, the user may say, “I need to meet Josh Peters tomorrow atthe same place where I was last Thursday at noon.” Device 110 sendsvoice data that reflects this input to NLP service 132. NLP serviceidentifies the phrase “at the same place where I was last Thursday atnoon” and, in response, determines where device 110 was located lastThursday at noon. NLP service 132 accesses location history data (storedin cloud 130 or stored on device 110 and sent to NLP service 132) anddetermines where device 110 was located last Thursday at noon. Thelocation history may indicate the name of a place or may consist ofgeographical coordinates. If geographical coordinates, then NLP service132 determines a name of the place or establishment that is located atthose coordinates. NLP service 132 causes that name to be associatedwith a task item for the task to meet Josh Peters on the date indicated.

Events that occur with respect to device 110 may also be used to createtask items. Such events may fall into one or more categories (or types)of context data described above, such as display data, presentationdata, and process data. For example, device 110 detects an incoming calland notifies the user of the call by causing a phone number or otheridentifying information about the call or caller to be displayed on ascreen of device 110. In addition to this information, the display mayinclude three selectable options: “Answer”, “Ignore”, and “Call BackLater.” If the user selects “Call Back Later”, then a task item iscreated where the task item identifies the caller and, optionally, atime of the call and/or a time to make a call to the caller. Also, thetask item may be automatically categorized as a task of type “To Call.”

Many of the examples herein regarding generating task items include auser providing voice or text input that includes details about a task.Another non-limiting example of how a task item may be generated is auser selecting (or highlighting) text that is displayed on a screen ofdevice 110. The selected text is considered context data. After the textis selected, the user may be presented with one or more options, one ofwhich is a “Remind” option which, when selected, causes a task item tobe generated. Task manager 112 generates the task item based on theinformation reflected in the selected text. Details of the task item maybe also determined from other context data, such as a time or event totrigger a notification of the task.

Virtual Dialogue

In some situations, NLP service 132 is unable to determine one or moredetails about a task based on input received from device 110 and thecontext associated with the input. Thus, in an embodiment, NLP service132 prompts a user of device 110 for further input to determine the oneor more details. The one or more details may pertain to any attribute ofa task item, such as the description of the task, the location of thetask, the location of a reminder (if any), or the time of the task.

For example, NLP service 132 receives, from device 110, voice data thatreflects a user's command to “Remind me to call Steve at 7.” NLP service132 may have access to information (e.g., an address book) aboutnumerous contacts, of the user, that have the name of Steve. Further,nothing in the address book can be used to disambiguate which of theSteve contacts to call. Therefore, NLP service 132 sends, to device 110,the following message to be displayed (or played audibly) by device 110:“Do you mean Steve Anderson, Steve Hanson, or Steve Jobs?” The user thenprovides, to device 110, voice or text input that indicates one of thethree Steve contacts. In response, device 110 sends the correspondingvoice or text data over network 120 to NLP service 132.

As another example, NLP service 132 receives, from device 110, voicedata that reflects a user's command to “I need to pick up bread at WholeFoods.” In response, NLP service 132 performs a lookup of the nearestWhole Foods stores to (a) the current location of device 110 or (b) theuser's home. There may be multiple Whole Foods stores that are neardevice 110's current location and near the user's home. Therefore, NLPservice 132 sends, to device 110, the following message to be displayedby device 110: “Which Whole Food's? The one on Almaden Rd, ChesterExpressway, or Green Street?” The user then provides, to device 110,voice or text input that indicates one of the three Whole Foods stores.In response, device 110 sends the corresponding voice or text data overnetwork 120 to NLP service 132.

As another example, NLP service 132 receives, from device 110, voicedata that reflects a user's command to “Remind me to text Jerry by 8.”In response, NLP service 132 determines, based on the voice data and thecontext of the input that Jerry is Jerry Wall, indicated in the user'scontact list (or address book). However, it is unclear whether the userintended 8 AM or 8 PM as the time to send an SMS message to Jerry.Therefore, NLP service 132 sends, to device 110, the following messageto be displayed by device 110: “Do you want to text Jerry Wall at 8 AMor 8 PM?” The user then provides, to device 110, voice or text inputthat selects one of the two times. In response, device 110 sends thecorresponding voice or text data over network 120 to NLP service 132.

Autocategorization of Task Items

In an embodiment, NLP service 132 determines, based on input from a userof device 110, one or more categories to associate with a task item. Theone or more categories may be one of many different categories, whichmay be virtually limitless. Non-limiting examples of categories withwhich a task item may be associated include things to purchase, thingsto do on vacation, things to do at work, and things to do while driving.Each category may be associated with a sub-category. For example, a“purchase category” may be divided into a grocery category indicatingitems to purchase at a grocery store, a book category indicating booksto purchase, and a music category indicating songs to purchase.

For example, a user may provide the following voice input to device 110:“Remind me to get milk.” Device 110 sends voice data that reflects thatinput to NLP service 132. NLP service 132 determines that a task itemshould be created and that “get milk” should be the descriptionassociated with the task item. NLP service 132 may also determine thatmilk is a grocery item and that the task item should be associated witha grocery category and/or a purchase category. Thus, NLP service 132 maysend, to task manager 112, category data that indicates one or morecategories with which the task item (whether created by NLP service 132,by a task service in cloud 130, or by task manager 112) should beassociated.

As will be described hereinafter, the one or more categories associatedwith each task item may be used to organize task items that belong tothe same category and display, on device 110, task items of the samecategory. This will allow a user of device 110 to view task items bycategory, in addition to or instead of by completion time, by creationtime, by trigger type (described hereinafter), by location, by type(e.g., reminder task v. non-reminder task), or by some other criterion.

II. Triggering Notifications of Task Items

As noted previously, a task item may be associated with one or moretriggering criteria (or triggers) that, when satisfied, causes anotification to be presented to a user of device 110 or some otheraction to be performed. When one or more triggering criteria of a taskitem are satisfied, a notification (or other action) is “triggered.”Non-limiting examples of triggering criteria include time, location,relative travel time, context triggers, and exogenous triggers, each ofwhich is described in more detail below.

Time Trigger

The time of a time trigger may be an absolute time, a relative time, arecurring time, or a symbolic deadline. An example of an absolute timeis Jun. 6, 2011, 9 AM Pacific Time. An example of a relative time is “10minutes before the Patriots-Jets football game.” An example of arecurring time is “Every Thursday at 10 AM.” An example of a symbolicdeadline is “end of business day”.

Location Trigger

According to an embodiment of the invention, the location of device 110is a triggering criterion associated with a task item. Such a triggeringcriterion is referred to herein as a “location trigger.” The location ofdevice 110 may be determined in one of many ways. For example, thelocation of device 110 may be automatically determined based on Wi-Fipositioning, cell positioning, and/or GPS (global positioning system)positioning. Device 110 may determine its current location with orwithout input from a service in cloud 130.

In an embodiment, a user may provide input that indicates a label to beassociated with a certain geographical location. For example, a user ofdevice 110 may speak the following sentence, “I am home” or “I am atWhole Foods.” NLP service 132 may then associate the word “home” orphrase “Whole Foods” with the current location of device 110, asdetermined based on one of the three positioning methods mentionedpreviously. This association of a word with a location may be laterleveraged to determine where “home” or “Whole Foods” is located.

A location trigger may not be associated with a specific geographiclocation or area. Instead, a location trigger may be associated with aplace that is not limited to a specific geographic location or area. Forexample, a location trigger of a task item may be “on the road” or“while driving.” Device 110 (or a process executing on device 110)determines that the current location of device 110 is on a freeway oranother busy road. Thus, this determination can be made regardless ofthe speed at which device 110 is moving or whether device 110 is pairedwith another device that would indicate that the user is traveling.Based on this determination, task manager 112 analyzes one or more taskitems to determine whether any task items are associated with the “onthe road” or “while driving” location trigger.

As another example, a location trigger of a task item may be the user'scar. Specifically, the user may have provided the following voicecommand: “Remind me to call my mom while driving.” NLP service 132analyzes voice data that reflects that command and determines that“while driving” refers to the user's car. The user's car may have aBluetooth-enabled component to allow device 110 to communicate with theuser's car. When device 110 comes into range of a Bluetooth signalpropagated by a Bluetooth-enabled component in the user's car, device110 determines that device 110 is located in (or at least near) theuser's car. In response to this determination, task manager 112 triggersthe location trigger of the task item. Task manager 112 causes areminder message to be displayed on device 110, where the remindermessage informs the user to call his mother. The user may then provide asingle tap or a voice response that causes a phone application executingon device 110 to initiate a call to a phone number associated with theuser's mom.

While establishing a connection (or “pairing”) with anotherBluetooth-enabled device is one example of pairing that can be used todetermine device 110's location, other types of pairings are possible.For example, device 110 may detect certain network data during theevening and morning hours. The network data indicates one or morenetworks to which device 110 may connect. The network data may includethe names of one or more networks or MAC addresses of one or morerouters. Device 110 may then determine that whenever that network datais detected, device 110 is considered to be at the user's home. Thus,actual pairing is not required since pairing entails the establishmentof a connection between device 110 and another device, such as a router.As another example, device 110 may detect a Wi-Fi signal on a train,subway, or bus. The Wi-Fi signal might indicate the type oftransportation that corresponds to the Wi-Fi signal. Thus, device 110might detect, based on the Wi-Fi signal, that its location is “on atrain,” “in a subway,” or “on a bus.” If a triggering criterion of atask item indicates one or more of these locations, then an actionassociated with the task item may be triggered. Further, such“transit-oriented” locations may also be considered to be associatedwith specific contexts (described in more detail below), such as “intransit” or “while traveling.” Thus, detection by task manager 112 ofsuch contexts may cause actions associated with certain task items to beperformed.

The foregoing examples of location triggers can be categorized as“arrival triggers,” such as are found in user input to “Remind me to doX when I arrive at Y.” Another type of location trigger is a “departuretrigger,” an example of which is found in the user command to “Remind meto do X when I leave work” or “ . . . when I leave here.” In anembodiment, in the departure trigger scenario, a minimum distance fromthe current location and the location of the departure is requiredbefore a particular departure trigger “fires.” Such a minimum distancemay be helpful to avoid the performance of corresponding actions whenthere are false starts.

Additionally, a location trigger may be one of multiple conditions thattrigger an action of a task item. Examples of user commands that includemultiple conditions include “Remind me to do X when I get home or at 8PM at the latest,” “Remind me to do X before 8 PM or when I leave,whichever is first,” and “Remind me to do X before 8 PM or while I amdriving, whichever is first.”

Travel Time Trigger

In an embodiment, the location of device 110 and a time associated witha task item is used to provide a notification to a user of device 110.Thus, while the time may be one of the one or more triggering criteriaassociated with the task item, the location of device 110 may not be, atleast explicitly so.

FIG. 3 is a flow diagram that depicts a process 300 for determining atime to provide a reminder to a user ahead of a scheduled time for atask, according to an embodiment of the invention. Process 300 may beperformed by one or more processes executing on device 110 or in cloud130. However, for ease of explanation, all the steps in process 300 areperformed by task manager 112.

At step 310, task manager 112 determines a current location of device110. At step 320, task manager 112 determines a location of adestination (or “destination location”) associated with (or identify by)a task item. At step 320, based on the distance between the twolocations, task manager 112 determines a “travel time,” or the time itmight take for the user of device 110 to travel to the destinationlocation. At step 330, task manager 112 determines a “difference time,”or the difference between the current time and the time triggeringcriterion associated with the task item. At step 340, if the travel timeis the same as or near the difference time, then task manager 112provides a notification to the user. This notification acts as areminder for the user to begin (if s/he has not already done so)traveling to the destination.

For example, a task item may be for a reminder to meet Sue at aparticular restaurant at 2 PM. Task manager 112 determines the locationof device 110 and the location of the particular restaurant. Thelocation of the particular restaurant may be determined by initiating,e.g., an Internet search and identifying the closest restaurant, withthe same name, to device 110's location. Alternatively, an address ofthe particular restaurant may already be stored in association with thetask item. Based on the distance between device 110's location and theparticular restaurant, task manager 112 determines how long it will takefor the user of device 110 to travel to the particular restaurant (or“travel time”). When the travel time is the same as or near (e.g.,within 10 minutes) the difference between the current time and the timetrigger (i.e., 2 PM), then task manager 112 causes, to be displayed ondevice 110, a message that indicates that the user should leave soon toarrive at the particular restaurant at 2 PM.

In an embodiment, the time of when to leave for a destination changesbased on the current location of device 110. For example, when thecurrent location of device 110 is at location A and the destination isat location B, task manager 112 determines that the user should begintraveling 50 minutes before the time of a scheduled task. However, inresponse to detecting that the current location of device 110 is now atlocation C, task manager 112 determines that the user should begintraveling 20 minutes before the time of the scheduled task. For example,a user of device 110 may be at home at the beginning of the day and taskmanager 112 determines that it will take 50 minutes to travel, from theuser's home, to the location of a dinner event in the evening. Later inthe day, the user of device 110 travels to work, which is closer to thelocation of the dinner event. In response to device 110 being at adifferent location, task manager 112 determines that it will take 20minutes to travel, from the user's work, to the location of the dinnerevent.

In an embodiment, the time of when to leave for a destination changesbased on current traffic information. For example, at 2:30 PM, taskmanager 112 determines that the time of when a user of device 110 shouldleave for a restaurant is 5:00 PM. However, due to a car accident on afreeway that the user can take to arrive at the restaurant, the trafficslows considerably. Task manager 112 determines, at 3:30 PM, that thetime of when the user should leave for the restaurant is 4:00 PM.

FIG. 4 is a view 400 of a travel reminder, according to an embodiment ofthe invention. View 400 is displayed by device 110. The travel reminderof view 400 contains six data items. The six data items include: (1) adescription 410 of the corresponding task (“pick up Chloe”); (2) a time420 of when to complete the task (“5:00 PM Today”); (3) an action 430 toperform when the user of device 110 should begin traveling to thedestination; (4) a reminder time 440 that indicates that the user wouldlike to be reminded of when the user should begin traveling to arrive atthe destination on time; (5) a start time 450 that indicates when theuser should begin traveling to arrive at the destination on time; and(6) a location 460 that indicates a name of the destination and anaddress of the destination. Another travel reminder that device 110displays may contain more or less data items.

The action associated with action 430 may be triggered (or performed) inresponse to task manager 112 determining that the current time(indicated at the top of travel reminder) equals the time indicated bystart time 450. In the illustrated example, action 430 is a map-relatedaction where task manager 112 causes a map to be generated at start time450 and displayed to the user of device 110. The map includes anindication of the address of location 460, an indication of the user'scurrent location, or both. Instead of automatically causing the map tobe displayed at start time 450, task manager 112 might first cause amessage to be displayed on device 110, wherein the message includes anoption to generate the map. If the user selects the option (e.g.,through voice input or tapping on the screen), then task manager 112causes the map to be generated and displayed.

A reminder setting may be in an “on” or “off” mode. In FIG. 4, remindertime 440 is in an “on” mode. If reminder time 440 is in an “off” mode,then the travel reminder of view 400 might not include reminder time 440or start time 450.

As indicated previously, task manager 112 might change start time 450 inresponse to changes in device 110's location. Thus, while start time 450may indicate “3:30 PM Today” when device 110 is located at the user'shome in the morning, start time 450 may indicate “4:20 PM Today” whendevice 110 is located at the user's work office in the afternoon.

In an embodiment, task manager 112 checks for changes in computed starttime 450 in response to significant changes in device 110's location.Significant changes in location may be determined as a result of otherevents that are already being computed. For example, device 110 mightalready process events when it transitions between cell towers, andthese events could trigger the re-computation of a change in locationand, therefore, in an updated start time 450. Other non-limitingexamples of events that indicate a potential significant change inlocation are changes in Wi-Fi signatures detected, the computation ofaccurate GPS locations for some other application (such as maps ornavigation), a power cycle event, turning on or off radios on thedevice, alerts based on accelerometer signals, and the receipt of textmessages or push notifications that contain location information.

In an embodiment, task manager 112 combines strategies for detectingsignificant event changes. For example, in a low power/low resolutionmode, task manager 112 only checks for significant location changesevery N minutes or only when some periodic computation occurs, such aschecking for incoming data. In a high power/high resolution mode, taskmanager 112 uses cell tower positioning and/or GPS. A combined strategymight run the low power solution by default and then invoke the highpower solution when the estimated start time is soon or when otherevents occur (for example, a change in Wi-Fi or Bluetooth signatures isdetected).

In an embodiment, a travel reminder or start time item in a travelreminder may be associated with one or more modes of transportation.Non-limiting examples of modes of transportation include driving a car,riding a bus, bicycling, and walking. A default transportation mode maybe driving a car. For example, task manager 112 may provide the optionfor a user to view start time 450 in a “car” mode, a “bus” mode, a“bike” mode, a “walking” mode, or multiple modes simultaneously.Depending on the current mode(s) selected for start time 450, the starttime may vary widely. For example, in FIG. 4, while start time 450indicates “4:20 PM Today” for a car mode, start time 450 may indicate“3:15 PM Today” for a bus mode, “3:45 PM Today” for a biking mode, and“11:30 AM Today” for a walking mode.

In a related embodiment, a task item is associated with both a locationand a date/time and a notification of the task may be triggered byeither the user (or, rather, the user's device) being at the location orby the date/time. For example, if the user's device is at the location,(either on the date or regardless of the date), then a notification istriggered. If the user has not arrived at the location on the dayindicated by the date (or at the location by the time), then the time isused as a “last resort” for triggering a notification.

Context Triggers

As described previously, time and location are examples of types oftriggering criteria associated with a task item. Another type oftriggering criteria associated with a task item is context. A “contexttrigger” refers to one or more characteristics of device 110 other thansimply the device 110's location. Thus, like context triggers, traveltriggers and travel time triggers also refer to one or morecharacteristics of device 110.

Context triggers may be categorized into one of two types: data-orientedcontext triggers and spatial-oriented context triggers. Non-limitingexamples of data and/or events on which data-oriented context triggerscan depend include the kind or type of data that device 110 isdisplaying on its screen (e.g., video), the specific application(s) ortype of application(s) that are currently executing on device 110 (e.g.,a texting application or a music application), the type of input thatdevice 110 is receiving from a user (e.g., voice, text, selections,button presses), the occurrence of (or a property of) a communicationevent (e.g., receiving or initiating a telephone call, text message, oremail, etc.), and the type of network connections available to device110 (e.g., Wi-Fi or cellular network).

For example, a user command that device 110 receives may be “I need toemail Bob when I am surfing the Internet.” The phrase “surfing theInternet” is presumed to mean that when the user is interacting with aweb browser on device 110, the context of device 110 (or of the user) is“when online.” In response to determining the context of the device orof the user, a reminder will be sent to the user to inform the user toemail Bob. Additionally another reminder may be provided to the user forany other task items that are associated with the “when online” contexttrigger.

As another example, a user command that device 110 receives may be“Remind me to email Jane Smith when I have a Wi-Fi connection.” Inresponse to device 110 detecting a Wi-Fi signal that does not require apassword or that requires a password accessible to device 110, taskmanager 112 causes a notification to be displayed on a screen of device110, where the notification indicates that Jane is to email Jane Smith.

Another type of data-oriented context trigger is a communication basedtrigger. Communication based triggers detect the occurrence of and/or aproperty of a communication event that occurs at or is detectable bydevice 110. Communication events include, for example, telephone calls,emails, text messages (e.g., SMS messages, Internet chat messages,IMESSAGE messages), videotelephony calls (e.g., FACETIME video chatsessions), and the like. These communication events can be used astriggering criteria for reminder notifications. In particular, bymonitoring various communication types, such as the examples listedabove, the usefulness and relevance of reminder notifications can beincreased. For example, triggering criteria can be selected to moreclosely match the user's intent in setting the reminder. For example,when a user asks to be reminded to mention something to a person “thenext time I talk to her,” it may more accurately reflect the user'sintent to trigger the reminder before or during any communication eventwith that person—regardless of the communication type—rather than justmonitoring for a communication event of a single type (e.g., a telephonecall).

In some implementations, communication based triggers are configured todetect the occurrence of a communication event with a certain person orgroup of people that the user identifies when creating the task item.For example, a user may issue a command to device 110 to “remind me toask my Mom about Thanksgiving plans next time I talk to her.” A taskitem can then be created that will detect the next communication eventbetween the user and his mother.

Several types of triggering criteria can be established forcommunication based triggers. For example, triggering criteria can bebased on the type of communication event (e.g., telephone calls, emails,etc.), the identity of a party to a communication event (e.g., a senderof an email, a participant in a telephone call), the initiator of acommunication event (e.g., who called whom), the content of acommunication event (e.g., whether the communication relates to aparticular subject or includes particular words), or any combination ofthese or other factors. Thus, the statement “next time I talk to her” inthe example above may be interpreted as a condition on the party to thecommunication event (i.e., the user's mother) and on the type ofcommunication event (i.e., that the communication type must include avoice communication component). As discussed below, that statement couldalso be interpreted without regard to the type of communication event,such that the reminder is triggered when any type of communication isdetected between the user and his mother. With respect to conditions onthe initiator of the communication, the statement “next time I talk toher” may suggest that the user did not intend to constrain thetriggering event based on who initiated the communication, and thereminder can be triggered regardless of who called whom, for example. Onthe other hand, the command “next time I call her,” may be interpretedsuch that the reminder is only triggered if the user initiates thecommunication with his mother. Further examples and explanations ofcommunication based triggers and various triggering constraints arediscussed below.

In some implementations, the specific type or types of communicationevents that trigger a reminder depend on the particular words in theuser input that established the reminder. For example, in order tocreate a task item with a communication based trigger, a user may issuethe following command to device 110 (e.g., as a voice input): “Rememberto text Mom when I am talking to my sister Sarah.” As noted above, insome implementations, the phrase “when I am talking to my sister Sarah”is presumed to mean that a reminder should be provided to the user whena voice communication (e.g., a phone call) is established with Sarah. Insome implementations, however, the command is interpreted to mean that areminder should be triggered when any type of communication event withSarah is detected, such as a phone call, an email, a text message, avideochat, etc. Thus, in some implementations, device 110 monitorscommunication events of multiple different types in order to determinewhether a triggering criterion has been met. Given the various ways inwhich people communicate, and the numerous communication types availableon the device 110, triggering a single notification with multiple typesof communications provides for flexible and convenient task reminders.Indeed, it would be a nuisance if a user had to create a different ruleor triggering criterion for each communication type, such as onereminder to be triggered based on an email, another based on a phonecall, another based on a text message, and so on, especially when all ofthe reminders would have identical content.

In some cases, though, users will wish to specify that only particularcommunication types are to trigger a particular notification. Forexample, a user may request a notification to be triggered “the nexttime I call James,” “the next time I email James,” or “the next time Itext with James,” etc. In these cases, tasks will be created withtriggers limited to phone calls, emails, or text messages, respectively.In some implementations, users may specify a particular group ofcommunication types to trigger a particular notification. For example, auser may request a notification for a task to be triggered “the nexttime I text or email with James.” Accordingly, this notification will betriggered by a text message or email communication event with a contactnamed “James.”

In some implementations, NLP service 132 interprets the particular termsin a task item creation command to determine what communication types tomonitor as triggering criteria for that task item. For example, in someimplementations, NLP service 132 interprets phrases such as “talk to,”“speak with,” “chat with,” and “hear from” as establishing a triggerbased on any communication type, and not only phone calls. Specifically,many users may understand these terms to relate to several types ofcommunication (e.g., phone, text, etc.), rather than to just a singletype (e.g., phone). Accordingly, when a command is received that usesthese terms, the triggering criteria will not be limited to only onecommunication type. Moreover, interpreting such terms to includedifferent communication types may reflect a user's intent moreaccurately: unless a user specifies a particular communication type(e.g., by reciting words such as “call” or “email” or “text”), it may bethat the user actually intends the reminder to be triggered upon theoccurrence of any of the various possible communication types. On theother hand, in some implementations, NLP service 132 interprets termssuch as “talk,” “speak,” and/or “chat” as establishing a triggerrequiring a voice-based communication event, such as a telephone call ora video chat.

In some implementations, where the user's command does not explicitlyrecite the communication types that are to trigger a notification, orwhen it is otherwise unclear from the user's command, device 110 (inconjunction with NLP service 132, discussed above) engages in a virtualdialogue with the user to disambiguate the user's input. In someimplementations, NLP service 132 prompts a user for further input todetermine additional details about the task item. For example, if a userrequests a reminder to be triggered “next time I talk to James,” device110 may respond by presenting to the user a question such as “do youmean when you are on the phone with James?” or “shall I also remind youwhen you email or text with James?” Such questions may be presentedaudibly, visually, or both. The user can then provide an answer to thequestion, which device 110 and/or NLP service 132 will interpret todetermine what types of communication events will trigger thatparticular notification. These and other techniques can be used todisambiguate other aspects of a user's input as well, such as ambiguousnames in user inputs (e.g., where multiple people named “James” appearin the user's contacts), and the like.

As noted above, communication based triggers can also depend on whichparty to a communication event initiated the communication. For example,a triggering criterion of “next time I call my wife” indicates that thenotification should be triggered only when the user initiates atelephone call with his wife, but not when the user's wife calls theuser. On the other hand, communication based triggers can also beagnostic to which party initiated the communication. For example, insome implementations, a triggering criterion of “next time I talk to mywife” will be satisfied regardless of whether the user called (oremailed, texted, etc.) his wife, or whether the wife called (or emailed,texted, etc.) the user.

In some implementations, communication based triggers are configured todetect the occurrence of communication events of a particularcommunication type without regard to whom the communication is with. Forexample, a user may issue the command “Remind me to call my mom nexttime I am on the phone.” The phrase “on the phone” can be presumed tomean that when the user is using device 110 as a phone, a notificationwill be sent to the user to inform the user to call his/her mom. Thisnotification will be triggered regardless of the other party to thephone call. In another example, the command “Remind me to email Bob nexttime I'm sending emails” results in a triggering criterion that will besatisfied when the user sends any email, regardless of the recipient.

In some implementations, communication based triggers are conditioned onthe content of a communication event. For example, a notification can betriggered only when a particular subject, topic, word, etc., is detectedin the content of a communication. The content of a communication eventcan be textual (e.g., the body of a text message or email), audio (e.g.,words spoken during a telephone call or video chat), or any othercontent (e.g., information relating to attachments to emails or textmessages). Various techniques may be used to determine whetherparticular content satisfies a triggering criterion. In someimplementations, natural language processing techniques are used todetermine subjects, topics, words, etc., in a communication. Forexample, NLP service 132 can analyze a transcript of a telephone call orthe body of an email to determine whether it satisfies the triggeringcriterion (e.g., whether the communication relates to “Project Bunny,”or discusses the stock price of Apple Inc.).

Communication based triggers can be used to trigger notifications ofvarious types. For example, notifications can include static contentand/or dynamic content. In some implementations, static content includestext or other fixed content (e.g., image, sounds, files, etc.) that isspecified by the user. For example, a task item may be created inresponse to the input “remind me to tell Sherri about the dinner planswhen she calls.” When Sherri calls, then, the notification will includetext such as “Tell Sherri about the dinner plans.” Dynamic ocntent, onthe other hand, includes information that is obtained, identified,updated, or retrieved at the time the notification is triggered. Forexample a task item may be created in response to a user input to “tellme the score of the last Packers game when I talk to Dad.” In thisexample, device 110 (or any appropriate combination of devices orsystems described herein) looks up the score of the most recent Packersgame when a communication event with the user's father is detected, andnotifies the user of the score. Other dynamic content that can be lookedup or searched includes emails, web content, stock prices, sportsscores/statistics, weather reports, news stories, and the like. Forexample, an input that creates a task item may include a request fordynamic content such as “find all recent emails from my boss when hecalls,” or “get the weather in Florida when my parents call,” or “searchthe web for news on Apple, Inc. when I'm texting with Tim.” Such contentis then presented to the user when the appropriate communication eventis detected.

In some implementations, the content (either static or dynamic content,as described above) is provided at the same device at which thecommunication event is detected. Continuing an example from above, whena phone call from Sherri is detected at a user's smart phone, the textof the reminder can be presented in a popup window (or spoken by anaudio output unit) on the smart phone. In some implementations, thecontent is provided at a different device. Continuing another examplefrom above, when a phone call from a user's boss is detected on a user'sphone, recent emails from the user's boss are displayed on a computerthat is separate from the phone (e.g., a laptop, desktop, or tabletcomputer).

Attention is directed to FIG. 18A, which is a flow diagram that depictsa method 1800 for triggering an action associated with a task, accordingto some implementations. In some implementations, the method 1800 isperformed by an electronic device (e.g., device 110) with a processorand memory storing instructions for execution by the processor. In someimplementations, the method 1800 is performed by device 110 inconjunction with one or more other electronic devices (e.g., devices inthe cloud 130). In some implementations, the method 1800 is performedentirely by an electronic device in the cloud 130.

A task item associated with a triggering criterion is received, whereinthe triggering criterion requires an occurrence of any of a plurality ofcommunication events including a telephone call (1802). For example, thetriggering criterion may be the occurrence of a telephone call, and aplurality of communication events are monitored in order to detect theoccurrence of the telephone call. In some implementations, thetriggering criterion requires that the communication event be acommunication with a specified person (1804).

In some implementations, at least two of the plurality of communicationevents are of different communication types (1806). For example, in someimplementations, communication events of different types (e.g., phone,email, text message, etc.) are monitored in order to determine whether acommunication event satisfying the triggering criterion occurs. In someimplementations, the different communication types include a telephonecall and an email (1808). In some implementations, the differentcommunication types include a telephone call and a text message (1810).In some implementations, the different communication types include atelephone call and a videochat (1812).

The occurrence of a first communication event of the plurality ofcommunication events is then detected (1814). In some implementations,the first communication event is a telephone call (1816).

Subsequently, it is determined that the triggering criterion issatisfied (1818). For example, in some implementations, detecting theoccurrence of a first communication event (1806) satisfies thetriggering criterion. In some implementations, other conditions mustalso be met in order to determine that the triggering criterion issatisfied. For example, as described below, a triggering criteria may besatisfied only upon a determination that content in a communicationevent relates to a predefined subject.

In response to determining that the triggering criterion is satisfied, anotification associated with the task item is caused to be presented toa user of the electronic device (1820). In some implementations, such aswhen the method 1800 is performed at least partially on device 110(e.g., a smart phone, laptop/desktop/tablet computer, etc.), causing thenotification to be presented to a user comprises any of displaying texton a screen of the device, outputting audio from an audio output unit,and the like. In some implementations, the notification includesinformation associated with the task item (1822). For example, if thetask item includes information (e.g., text) that is to be presented as areminder (e.g., “ask Mom about Thanksgiving plans”), that informationcan be presented to the user—either visually, audibly, or both—when thetriggering criterion is satisfied. In some implementations, whereinformation is presented to the user audibly, it is presented such thatonly the user can hear it. For example, a voice output including thewords “ask Mom about Thanksgiving plans” may be presented to a userduring a telephone call with his mother such that only the user can hearthe output.

In some implementations, information associated with the task itemincludes dynamic information, as described above. The dynamic content isthen obtained, retrieved, or identified when the notification ispresented to the user. For example, the task item may be associated witha request to “show me the stock price for Apple Inc. when Tim calls.”When the triggering criterion is satisfied for this task item (e.g.,“Tim” called the user), the current stock price for Apple Inc. isretrieved and presented to the user as part of the notification.

Step (1820) of method 1800 continues on FIG. 18B. In someimplementations, the first communication event is a text message, andcausing the notification to be presented comprises causing an affordancerelating to the first task item to be displayed on the electronic devicenear a representation of the text message (1824). In someimplementations, the affordance is a selectable element displayed on atouch screen of an electronic device (e.g., a smart phone). Theaffordance may be configured to cause an action associated with the taskitem to be performed when the affordance is selected or otherwiseactivated. For example, in some implementations, the affordanceinitiates a communication event (e.g., a phone call), presents a drafttext message or email to the user, or the like.

In some implementations, the task item includes information thatdetermines whether an affordance is to be displayed, and what actionshould be taken in response to a selection of the affordance. Forexample the task item may have been created in response to a user inputto “send a text to Mom when I talk to Sarah.” When a communication eventwith Sarah is detected, a button may be presented in association withthe text “Send Text to Mom.” In some implementations, selection of thebutton causes a predefined text message to be sent to the user's mother,or causes a text message editing application or region to appear so thatthe user can compose or edit a text message.

In some implementations, the first communication event is an email, andcausing the notification to be presented comprises causing an affordancerelating to the first task item to be displayed on the electronic devicenear a representation of the email (1826). In some implementations, theaffordance is a button that, when selected by the user, causes an actionassociated with the task item to be performed (1828).

In implementations where a draft communication (such as a draft email,text message, or any other draft communication that includes fields) ispresented to a user, one or more of the fields of the draft arepre-populated with information from or related to the task item. Forexample, if a draft text message is presented to the user in response toa task item to “send a text message” to a recipient, the “to” field maybe pre-populated with a phone number, email address, or other identifierof the recipient. Moreover, the “message” field (e.g., for the body ofthe text message) may even be pre-populated. As another example, if adraft email is presented to the user, the “to” field may bepre-populated with the email address of the intended recipient, and the“subject” and “message” fields may also be pre-populated if suchinformation is available. The name and/or contact information of therecipient and/or the content for the “subject” and “message” field of atext message or email may be included in the task item. As noted above,such a draft may be presented to a user in response to the userselecting an affordance presented in a notification item, or in responseto a triggering criterion being satisfied and without the userseparately selecting an affordance.

In some implementations, the first communication event is a telephonecall, and causing the notification to be presented comprises causing anaudio output relating to the first task item to be produced by theelectronic device (1830). The audio output can be any audio, includingsynthesized or recorded voice output, tones or beeps, music, and thelike. In some implementations, the audio output is produced during atelephone-notification period (e.g., while the telephone is ringing)(1832). For example, the audio output (e.g., a synthesized voice output)is produced during the time in which a phone is ringing to alert a userto an incoming telephone call. In some implementations, the ringing ofthe incoming call is muted or its volume is reduced so that the audiooutput can be heard by a user. In some implementations, the audio output(e.g., a synthesized voice output) is produced during a ringing periodof an outgoing call. For example, a synthesized voice saying “don'tforget to ask about Thanksgiving plans” may briefly interrupt theringing sound of an outgoing call when it is detected that the user iscalling his mother. In some implementations, the ringing is muted whilethe audio is being presented to the user. In some implementations, thevolume of the ringing is lowered, but is still audible, while the audiois being presented to the user. In some implementations, if thenotification period ends while audio is being presented, such as if therecipient picks up the phone, the audio output is terminated.

In some implementations, once a user initiates a communication eventthat satisfies a triggering criterion, actual execution of thecommunication event can be delayed until the audio output is presented.For example, if the user initiates a telephone call that satisfies atriggering criterion (e.g., by entering a phone number and pressing a“call” button or touchscreen element), the audio output is presented tothe user before the phone call is actually placed. Accordingly, theaudio output (e.g., “don't forget to ask about Thanksgiving plans”) canbe presented to the user in its entirety without being interrupted bythe recipient answering the phone, and without interrupting the user'sconversation.

In some implementations, the audio output is produced during a telephoneconversation between a user of the electronic device and a participantof the telephone call (1834). For example, an audio output, such as atone, beep, or the like, can be output during a telephone conversationto alert a user that a notification has been activated. This can promptthe user to look at a screen of the telephone. Or, a synthesized voicecould be output by the device, during the telephone conversation for theuser to hear. In some implementations, the audio output is configured sothat the other party to the telephone conversation does not hear theoutput. In some implementations, the audio output is produced at a lowervolume than a voice of the participant of the telephone call (1836).

Method 1800 continues on FIG. 18C. In some implementations, the firstcommunication event includes content, and the triggering criterionrequires that the content relate to a predefined subject (1838). Forexample, the first communication event may contain voice content (in thecase of a telephone call or video chat, for example), textual content(in the case of an email or text message, for example), attached filecontent, image content, video content, and the like. Triggering criteriafor a notification can be conditioned on one or more properties of suchcontent, such that a notification is presented to a user when acondition relating to the content is satisfied. For example, a task itemmay be configured to present a notification to a user “the next timeJack emails about Project Bunny.” Accordingly, device 110 (and/or othersuitable devices or services) determines whether emails received fromJack relate to “Project Bunny,” and if so, the triggering criterion isdeemed satisfied. Returning to FIG. 18C, in some implementations, wherethe first communication event includes content, and the triggeringcriterion requires that the content relate to a predefined subject(1838), it is determined that the content relates to the predefinedsubject (1840). In some implementations, determining that the triggeringcriterion is satisfied (e.g., at step (1818)) is based at least in parton the determination that the content relates to the predefined subject(1842). In some implementations, the determination that the contentrelates to the predefined subject is performed using natural languageprocessing techniques (e.g., performed by NLP service 132, discussedabove).

In some implementations, the triggering criterion requires that thecontent of a first communication event relate to a predefined subject,and the first communication event is a telephone call (1844). In someimplementations, a text string corresponding to one or more utterancesspoken during the telephone call by a participant in the telephone callis obtained (1846). In some implementations, it is determined whetherone or more words in the text string relate to the predefined subject(1848). In some implementations, NLP service 132 converts voice dataobtained from one or more of the participants in a telephoneconversation into text. The text is then processed (e.g., by NLP service132 and/or device 110) to determine whether one or more words in thetext string relate to the predefined subject. For example, if atriggering criterion for a notification is “when I talk to Jack aboutProject Bunny,” a conversation between the user and Jack can be analyzedto determine whether either party says the words “Project Bunny,” orsays other words that may be indicative of that subject.

Attention is directed to FIG. 19, which is a flow diagram that depicts amethod 1900 for triggering an action associated with a task, accordingto some implementations. In some implementations, the method 1900 isperformed by an electronic device (e.g., device 110) with a processorand memory storing instructions for execution by the processor. In someimplementations, the method 1900 is performed by device 110 inconjunction with one or more other electronic devices (e.g., devices inthe cloud 130). In some implementations, the method 1900 is performedentirely by an electronic device in the cloud 130.

A task item associated with a triggering criterion is provided, whereinthe triggering criterion requires an occurrence of any of a plurality ofcommunication events, and wherein at least two of the plurality ofcommunication events are of different communication types (1902). Insome implementations, the triggering criterion requires that thecommunication event be a communication with a specified person (1904).In some implementations, the first communication event is associatedwith the specified person when the specified person initiates the firstcommunication event or receives the first communication event from theuser (1906). For example, a communication event may be associated with aspecified person regardless of who initiated the communication event(e.g., regardless of who called whom, who emailed whom, etc.). In someimplementations, the different communication types include a telephonecall and an email (1908). In some implementations, the differentcommunication types include a telephone call and a text message (1910).In some implementations, the different communication types include atelephone call and a video chat (1912).

The occurrence of a first communication event of the plurality ofcommunication events is detected (1914). For example, an incoming oroutgoing telephone call, email, text message, or the like is detected.It is determined that the triggering criterion is satisfied (1916). Forexample, in some implementations, just detecting the occurrence of afirst communication event (1914) satisfies the triggering criterion. Insome implementations, however, other conditions must be met in order todetermine that the triggering criterion is satisfied. For example, asdescribed above, a triggering criteria may be satisfied only upon adetermination that content in a communication event relates to apredefined subject. In response to determining that the triggeringcriterion is satisfied, a notification associated with the task item iscaused to be presented to a user of the electronic device (1918). Insome implementations, the notification includes information associatedwith the task item (1920). For example, if the task item includes text(e.g., information) that is to be presented as a reminder (e.g., “askMom about Thanksgiving plans”), that text can be presented to theuser—either visually, audibly, or both—when the triggering criterion issatisfied.

Attention is directed to FIG. 20, which is a flow diagram that depicts amethod 2000 for triggering an action associated with a task, accordingto some implementations. In some implementations, the method 2000 isperformed by an electronic device (e.g., device 110) with a processorand memory storing instructions for execution by the processor. In someimplementations, the method 2000 is performed by device 110 inconjunction with one or more other electronic devices (e.g., devices inthe cloud 130). In some implementations, the method 2000 is performedentirely by an electronic device in the cloud 130.

A task item associated with a triggering criterion is provided, whereinthe triggering criterion requires an occurrence of a telephone call atan electronic device (2002). In some implementations, the triggeringcriterion requires that the communication event be a communication witha specified person. The occurrence of the telephone call is detected(2004). In response to detecting the occurrence of the telephone call, anotification associated with the task item is caused to be presented toa user of the electronic device, wherein the notification is separatefrom a telephone call notification (2006). In some implementations, thenotification includes text that is displayed on a screen of a smartphone during or before a telephone call notification (e.g., a ringtone,vibration, or other call notification technique). In someimplementations, the notification includes audio content that ispresented to the user during or before a telephone call notification.

Referring now to spatial-oriented context triggers, non-limitingexamples of spatial-oriented context triggers include the speed at whichdevice 110 is moving (e.g., over 30 mph indicating driving, or less than3 mph indicating walking), a direction (absolute or relative) at whichdevice 110 is moving, and a set of movements of device 110 (e.g., shortvertical movements while moving continuously in a horizontal direction).In other words, device 110 may be configured to detect how device 110 ismoving through space.

For example, device 110 (or rather a process executing on device 110)determines, based on detecting changes in its location over a period oftime, that device 110 is moving at 60 mph. Based on this information,device 110 determines that the device's context is “while driving” or“on the road.” Task manager 112 analyzes one or more task items todetermine whether any task items are associated with a “while driving”or “on the road” context trigger. If a task item is associated with a“while driving” or “on the road” context trigger, then an action (e.g.,displaying a notification) associated with the task item is performed.

As another example, device 110 determines, based on detecting changes inits location over a period of time, that device 110 is moving towardshis home over a certain period of time (e.g., 5 minutes). Based on thisinformation, device 110 determines that the context is “on my way home.”Task manager 112 analyzes one or more task items to determine whetherany task items are associated with a “on my way home” context trigger.If a task item is associated with a “on my way home” context trigger,then an action (e.g., displaying a notification) associated with thetask item is performed.

As another example, device 110 includes an accelerator that detectscertain repetitive movements. Device 110 may determine, based on theserepetitive movements over a period of time, that the user of device 110might be running at a slow pace. Based on this determination, device 110determines that the context is “while jogging.” Task manager 112analyzes one or more task items to determine whether any task items areassociated with a “while jogging” or “while walking” context trigger. Ifa task item is associated with a “while jogging” or “while walking”context trigger, then an action (e.g., displaying a notification)associated with the task item is performed.

As another example, device 110 might detect that it has not moved for aperiod of time (e.g., 3 hours). A user of device 110 might be interestedin being alert and non-movement of device 110 might indicate that theuser is asleep. Thus, the user might issue the command, “Alert me if thephone doesn't move for 3 hours.”

In addition to data-oriented and spatial-oriented triggers, other kindsof triggers may be based on any sensor on device 110. Device 110 mayinclude multiple sensors, such as temperature sensors and light sensors.For example, device 110 might include a thermometer for detecting theoutside temperature or an internal temperature of device 110. Thus, auser of device 110 might issue the command, “Remind me to call Haroldwhen it reaches 100 degrees.”

Exogenous Triggers

Another type of triggering criteria that may be associated with a taskitem is exogenous criteria. An “exogenous trigger” is a triggeringcriterion that depends on one or more factors that exist outside andseparate from device 110 and the user of device 110. Such factors may beconsidered “events” that occur with respect to devices other than device110 or with respect to data that is stored on one or more devices otherthan device 110. Non-limiting examples of exogenous triggers includesocial location, social proximity, standing queries, and local events.

An example of a social location trigger is when a friend or associate ofthe user of device 110 arrives or leaves a certain location. Forexample, a user command that initiated the creation of a task item mayhave been “Notify me if Sarah leaves the mall.” Thus, the location ofSarah (or Sarah's mobile device) is an essential factor in setting offthis type of exogenous trigger. Specifically, task manager 112determines the current location of Sarah's device. The current locationof Sarah's device may be provided by a cloud service (e.g., in cloud130) to which both Sarah's device and device 110 are subscribed. Device110 receives, from the cloud service, updates as to the location ofSarah's device. Task manager 112 uses that location information todetermine whether the social location trigger should be activated. Asimilar user command is “Remind me when my daughter gets home.”

An example of a social proximity trigger is when a friend or associateof the user of device 110 is within a certain distance of the user (ordevice 110). For example, a user command that initiated the creation ofa task item may have been “Remind me to call George when he is within100 feet of me.” Thus, the location of George (or George's mobiledevice) is an essential factor in setting off this exogenous trigger.Specifically, task manager 112 or another process executing on device110 compares the current location of device 110 with the currentlocation of George's device to determine the distance that separates thetwo devices. Alternatively, George's device may transmit its location toa cloud service to which both George's device and device 110 aresubscribed. Device 110 receives, from the cloud service, updates as to adistance between George's device and device 110. Task manager 112 usesthat distance information to determine whether the social proximitytrigger should be activated.

An example of a standing query trigger is when a webpage mentions aparticular term or phrase, such as a company name. To detect this, astanding query is generated and issued continuously (e.g., once a day).For example, a user command that initiated the creation of a task itemmay have been “Tell me when cnn.com mentions Berkman Industries.” Taskmanager 112 or another process executing on device 110 issues a searchquery (e.g., to a search engine) and receives results. When task manager112 determines that the results include a webpage from cnn.com thatincludes the name “Berkman Industries,” task manager 112 provides anotification to the user of device 110.

An example of a local event trigger is when a certain local eventoccurs. To detect this, task manager 112 receives data from an onlineservice. Task manager 112 (or a task service in cloud 130) mayperiodically send a request to the online service (via one or morecommunication protocols). Alternatively, task manager 112 may subscribewith the online service to receive information about certain events. Forexample, a user command that initiated the creation of a task item mayhave been “Tell me when Beatles tickets go on sale at Shoreline.” Inresponse, task manager 112, another process executing on device 110, orNLP service 132 sends a subscription request to an online ticket serviceto receive a notification when Beatles tickets for a performance atShoreline Amphitheatre become available for purchase. When task manager112 is determines Beatles tickets are available for purchase, taskmanager 112 provides a notification to the user of device 110.

As another example, a user might be interested in knowing when the surfis up. Thus, the user might issue the command, “Remind me an hour beforethe surf is up.” Task service 112 (or a task service in cloud 130) mightregularly issue a query of a surfing site or might subscribe for alertsfrom the surfing site.

Based on the foregoing, the types and examples of exogenous triggers arevirtually endless. As long as task manager 112 (or a task service incloud 130) can make a determination about an event that occurs separatefrom device 110, that event can be used to trigger the performance of anaction associated with a task item.

III. Consuming Task Items (Active Payloads)

A task item is “consumed” when an action associated with the task itemis performed. Such an action may be a notification that is displayed (orplayed, if the notification is an audio notification) on device 110. Inaddition to or instead of providing a notification to a user of device110, other possible actions include initiating a phone call or a searchquery, sending an HTTP request (that includes a Uniform ResourceLocation (URL)), sending an email or a text (SMS) message, causing anapplication to execute, and causing a purchase to be made on the user'sbehalf. Such actions that can be associated with task items are referredto as “active payloads.” The processing of an active payload causes someaction to be performed, whether by task manager 112 or by anotherprocess, whether local or remote to device 110. In other words, insteadof simply notifying the user of a task associated with a task item, taskmanager 112 (or a service in cloud 130) can automate the action part ofthe task item.

As alluded to above, causing an action to be performed may involve taskmanager 112 causing another application or process to perform theaction. The calling or invoking of the other application (e.g., via anAPI of the other application) may be performed with or without furtherinput, as indicated in the following examples.

The types of “other” applications can vary greatly. Non-limitingexamples of applications that might be available on device 110 include aphone application, an email application, a Web browser, a music playerapplication, a media player application, a music download application,an image processing application, a geopositioning application, acontacts application, an SMS application, a video game application, anda text processing application.

For example, a user of device 110 says aloud, “Remind me to call herback this afternoon.” This voice input is converted into voice data thatdevice 110 sends (along with context data) over network 120 to NLPservice 132. NLP service 132 analyzes the voice data and the contextdata to determine that “her” refers to Marilyn Merlot. NLP service 132determines that “afternoon” is 2 PM (whether based on context data, apre-defined setting, or prior history) and determines a phone number forMarilyn Merlot based on a contacts list (or address book), associatedwith the user, that includes one or more phone numbers for MarilynMerlot. The contacts list may be stored on device 110 or in cloud 130.NLP 132 sends, to task manager 112 (or to a task service in cloud 130),reminder data used to generate a task item. The reminder data includesthe date of “Today”, time of 2 PM, and an instruction to call MarilynMerlot using a particular phone number. When task manager 112 determinesthat the current time is 2 PM, task manager 112 may cause a message tobe displayed that prompts the user to call Marilyn Merlot. The messagemay include a “Later” button and a “Call Now” button. If the userselects the “Later” button, then task manager 112 will send the messageagain later in the afternoon (e.g., in 1 hour). If the user selects the“Call Now” button, then task manager 112 initiates a call to MarilynMerlot. This initiation may involve task manager 112 making an API callto a phone application (not shown) executing on device 110 and passingthe phone number as an argument of the API call. The phone applicationthen uses the phone number to call a device associated with the phonenumber.

As another example, a user of device 110 says aloud, “Text Lindsay thatI love her at 5 o'clock.” This voice input is converted into voice datathat device 110 sends over network 120 to NLP service 132. NLP service132 analyzes the voice data to determine that a cell phone number ofLindsay is necessary and that “5 o'clock” refers to 5 PM of the currentday. Task manager 112 (or a task service in cloud 130) creates a taskitem that includes the following data items: (1) a completion time of 5PM today, (2) an action of sending a text (or SMS) message, (3) a numberof Lindsay's cell phone, and (4) a text string of “I love you” that willbe part of the text message. In response to determining that the currenttime is 5 PM, task manager 112 analyzes the task item to determine theaction that needs to be performed. Task manager 112 then causes a textmessage that includes the text string associated with the task item tobe sent to Lindsay's cell phone. This step may comprise task manager 112invoking an API call of a texting application (not shown) executing ondevice 110, where the text string (“I love you”) is an argument of theAPI call.

As another example, a user of device 110 says aloud, “Show me directionson how to get to Rachel's Restaurant in San Jose when I leave theoffice.” This voice input is converted into voice data that device 110sends over network 120 to NLP service 132. NLP service 132 analyzes thevoice data to determine that a cell phone number of Lindsay is necessaryand that “5 o'clock” refers to 5 PM of the current day. Task manager 112(or a task service in cloud 130) creates a task item that includes thefollowing data items: (1) a location trigger of leaving the user'soffice and (2) an action of displaying instructions (and, optionally, amap) on how to arrive at Rachel's Restaurant from the user's office. Inresponse to determining that the user of device 110 has left his/heroffice, task manager 112 analyzes the task item to determine the actionthat needs to be performed. Task manager 112 then causes (withoutfurther input from the user) a travel directions request to be sent to atravel directions service. The travel directions request includes thename of the restaurant, any address information of the restaurant, orboth. The travel directions service may be hosted on device 110 or onanother device (not shown).

As another example, a user of device 110 says aloud, “Order a cheeseonly pizza at Pizza Heaven in San Jose, home delivered, 30 minutesbefore the Bulls-Pacers game starts.” This voice input is converted intovoice data that device 110 sends over network 120 to NLP service 132.NLP service 132 analyzes the voice data to determine that a Bulls-Pacersgame starts at 6 PM local time; thus, the time trigger is 5:30 PM localtime. NLP service 132 also determines that Pizza Heaven in San Joseallows online ordering. Task manager 112 (or a task service in cloud130) creates a task item that includes the following data items: (1) atime trigger of 5:30 PM and (2) an action of ordering a cheese onlypizza from Pizza Heaven with home delivery as an option. In response todetermining that the current time is 5:30 PM, task manager 112 (or atask service in cloud 13) analyzes the task item to determine the actionthat needs to be performed. Task manager 112 then causes a pizza orderrequest to be sent to Pizza Heaven's online ordering service. The pizzaorder request includes the pizza type of cheese only, the deliveryoption of home delivery, and the user's home address. The pizza orderrequest may be in the form of an API call to the online orderingservice, where arguments of the API call include indications of cheeseonly topping, home delivery, and the user's home address. Alternatively,before causing the pizza order required to be sent, task manager 112 mayformulate a message that is displayed on (or played by) device 110,where the message informs the user about this task. If the user providesaffirmative input, then task manager 112 causes the pizza request orderto be sent. If the user provides negative input, then no pizza requestorder is sent.

As another example, a user of device 110 says aloud, “Play my classicalstation on Pandora at 3 PM tomorrow.” The time of “3 PM tomorrow”coincides with the end of a scheduled exam for the user. This voiceinput is converted into voice data that device 110 sends over network120 to NLP service 132. NLP service 132 analyzes the voice data todetermine a date and time that the intended action is going to beperformed. Task manager 112 (or a task service in cloud 130) creates atask item that includes the following data items: (1) a time trigger of3 PM with a date that identifies the following day and (2) an action ofplaying a classical “station” of the user's Pandora music application,where the classical station was established by the user and associatedwith the user's Pandora account. In response to determining that thecurrent time is 3:00 PM on the proper date, task manager 112 (or a taskservice in cloud 13) analyzes the task item to determine the action thatneeds to be performed. Task manager 112 then causes the Pandora musicapplication (not shown) to begin executing on device 110 and to “play”the user's classical station. Task manager 112 may cause the classicalstation to play by invoking an. API call to the Pandora musicapplication, where an argument of the API call includes an indication ofa classical station.

In some embodiments, active payloads are processed automatically basedon one or more triggering events (e.g., time, location, etc.) withoutproviding reminders, notifications, or otherwise requesting furtherpermission from the user. The device 110 can thus automatically takeactions on behalf of the user without the user needing to providefurther input. For example, task items can cause the device 110 to senda pre-written email or text message to a recipient in response to atriggering event. A user could say to the device 110 “tell my wife I'malmost there when I get off the freeway.” The device 110 (and/or anyassociated services such as NLP service 132) can create a task item withan active payload that will send a message (such as a text message oremail) saying “I'm almost there” to the user's wife when the userreaches a particular location (e.g., an exit ramp near the user'shouse). In some embodiments, a recording of the user's voice inputcorresponding to the message “I'm almost there” can be delivered to therecipient via voicemail, or as an attachment to an email or textmessage. Also, other triggering events can also cause active payloads tobe processed automatically and without user intervention, such as time,date, receipt of messages from others, etc.

The active payloads described above enable the device 110 to initiatecertain actions based on a triggering criterion of a task item. Forexample, if a task item is a reminder to call a co-worker at a 2:00 PM,the device 110 may automatically prompt the user with a “Call Now”button at 2:00 PM. In some cases, though, it is beneficial to configurethe task items so that the active payload can be processed without atriggering event. Specifically, a user may create task items that do nothave triggers (such as a time or location trigger), or a user may wishto complete a task item before its triggering condition is met. Toaccommodate this, in some embodiments, an action associated with a taskitem is performed in response to a user selecting the task item itself.By linking the active payload to the task item (or an input associatedwith the task item), the active payload can be processed at the user'srequest without requiring other triggering conditions to be met.Further, it allows a user who is viewing a task list to take advantageof active payloads by simply selecting the task item from the list attheir own convenience. For example, a user may view a task list toreview his outstanding tasks, and see a task item that he couldconveniently complete right away. The user can then simply select thetask item (e.g., the description “call Greg Almond”) in order to processthe active payload (e.g., initiate a phone call to Greg Almond) andcomplete the task item.

In order to be selectable by a user, tasks list items may be associatedwith an activation region that, when selected by a user, causes theaction associated with the task to be performed. Activation regions areany area of a graphical display that can be selected by pressing,clicking, touching, or otherwise selecting the area of the display. Insome embodiments, the activation region coincides with or overlapsgraphical or textual elements that are displayed on a screen (e.g., atouch screen) of the device 110. In some embodiments, the activationregion coincides with or overlaps a title or description of the taskitem, such that when a user selects the displayed text of the title ordescription, the device 110 performs an action associated with that taskitem. Task items that may include selectable text in accordance withsome embodiments are shown in FIG. 6.

In some embodiments, the activation region coincides with or overlaps agraphical element that is displayed on the touch screen in associationwith the task list item. Graphical elements may be displayed next to, inline with, above, below, or otherwise near the task list item,indicating to the user that the action associated with that task listitem will be performed upon selection of the graphical element.

Examples of graphical elements displayed in conjunction with task itemsare shown in FIG. 6. For example, graphic 630 includes a graphicaldepiction of a telephone handset, which can be made user-selectable byoverlaying an activation region on the graphic 630. When the graphic 630is selected by a user (e.g., by pressing, tapping, touching, etc.), thedevice 110 will perform actions associated with initiating a telephonecall to “John Appleseed.” For example, the device 110 may place atelephone call to a phone number associated with John Appleseed.

Similarly, graphic 632 includes a graphical depiction of a letter, whichcan be made user-selectable by overlaying an activation region on thegraphic 632. When the graphic 632 is selected by a user (e.g., bypressing, tapping, touching, etc.), the device 110 will perform actionsassociated with sending a text or email message to the recipient “PabloMarc.” For example, the device 110 may display a prompt with a textinput region to a user, in which the user can compose a messageaddressed to Pablo Marc.

In some embodiments, a user can select a task item using a voice input.Voice activation may be used alone or in conjunction with activationregions as described above. In some embodiments, a task item is selectedwhen the device 110 detects a spoken utterance that matches a title,description, or indicator of the task item. For example, a user viewingthe task list in FIG. 6 may say aloud “reply to Pablo Marc,” which wouldcause the device 110 to process an active payload associated with thattask item. In another example, a user may say aloud “Call John,” and thedevice 110 will recognize that this utterance corresponds to the taskitem “call John Appleseed” and process an active payload associated withthat task item.

In some embodiments, when a task item is selected, an action associatedwith that task is performed without further user intervention. Forexample, if the user selects the task item “call John Appleseed,” thedevice 110 will initiate a telephone call to John Appleseed. In someembodiments, selection of a task item causes a prompt to be displayed tothe user, asking the user for confirmation that the device 110 shouldperform an action associated with that task item. This confirmation orpermission step can help prevent inadvertent actions if a user selects atask list item by mistake. For example, upon selection of the task item“call John Appleseed,” the device 110 may display a prompt withselectable options of “Call Now” or “Cancel.” Similarly, upon selectionof the task item “reply to Pablo Marc,” the device 110 may provide anempty text composition area (where the user can compose the body of anemail or text message to be sent to Pablo Marc) and selectable optionsof “Send” or “Cancel.”

Attention is directed to FIG. 17, which is a flow diagram that depicts amethod 1700 for initiating actions associated with task items, accordingto an embodiment of the invention. In some embodiments, the method 1700is performed by an electronic device (e.g., device 110) with a processorand memory storing instructions for execution by the processor. Themethod 1700 includes receiving, from a user, input that specifies one ormore attributes for a task (1702). (See also FIG. 2, step (210), andattendant discussion.) In some embodiments, the input is based on voiceinput that was received at a handheld device (e.g., the device 110). Insome embodiments, the input is based on text input that was received ata handheld device through a text-input device (e.g., physical keyboard,touch screen keyboard, etc.). In some embodiments, the input initiatesthe creation of the task item, and includes data that identifies anaction (to be performed by the device) associated with the task. Forexample, inputs may include the words “send message,” “email,” or“call,” which the device 110 could recognize as identifying theparticular action that should be associated with that task. The method1700 further includes generating a task item based on the input,including identifying an action to be performed by the electronicdevice, wherein the action is associated with the task (1704). (See alsoFIG. 2, step (220), and attendant discussion.)

In some embodiments, the task item is not associated with a triggeringcriterion (e.g., a time or location trigger) that determines when totrigger a notification to complete the task, or when the action is to beperformed. The task item “call Greg Almond” in FIG. 6 illustrates a taskitem that is not associated with a triggering criterion.

The identified action can be any action that the device 110 can perform,including those actions discussed above in reference to active payloads.In some embodiments, the action is causing an email or a text message(e.g., including a previously drafted message) to be sent over a networkto a recipient that is associated with the task item. In someembodiments, the action is displaying a text composition area. In someembodiments, text input by a user into the text composition area is thensent as an email or a text message over a network to a recipient that isassociated with the task item. In some embodiments, the text compositionarea is associated with an email address or telephone number of arecipient associated with the task item. In some embodiments, the actionis initiating a phone call to a device associated with a phone numberthat is associated with the task item. In some embodiments, the actionis initiating a search based on a search query or URL that is associatedwith the task item.

In some embodiments, causing an action to be performed (such as one ofthe actions described above) comprises a first process causing a secondprocess to perform the first action. In some embodiments, the firstprocess causing the second process to perform the action comprises thefirst process invoking an API call of the second process, wherein theAPI call comprises one or more attributes associated with the task item

The method 1700 further includes displaying, in a task list, the taskitem and an associated activation region (1706). (See also FIGS. 6 and8.) In some embodiments, the activation region overlaps a title of thetask item, such that the title of the task item is selectable by a user.In some embodiments, the activation region overlaps a description of thetask item, such that the description of the task item is selectable by auser. In some embodiments, the activation region overlaps a graphicalobject, which may be displayed next to or near a title and/or adescription of the task item, such that the graphical object isselectable by a user.

The method 1700 further includes receiving a user selection of theactivation region (1708). The user may select an activation region byclicking, touching, pressing, and the like. In some embodiments, themethod 1700 includes after receiving a user selection of the activationregion and prior to performing the action, displaying a promptrequesting permission to perform the action (1710), and receiving apermission input from the user (1712). Requiring a user to read and/orrespond to a prompt with permission to proceed can help prevent thedevice 110 from taking actions that the user did not intend.

The method 1700 further includes performing the action in response toreceiving the user selection of the activation region (1714). Someexamples of actions that may be performed are described above withreference to step (1704).

In some embodiments, the method 1700 further includes, in response toperforming the action, marking the task item as complete (1724).

Automated Task Completion

In an embodiment, task manager 112 (or a task service in cloud 130)“marks” the task item as complete in response to detecting that a taskitem is consumed. In other words, a task item may be associated with acomplete or an incomplete status. Task manager 112 may provide aninterface for a user to view task items managed by task manager 112 anddetermine whether a task item is complete or not. Task manager 112 mayprovide an option for a user of device 110 to view all completed taskitems. The completed task items may be ordered based on when the taskitems were created, consumed (or completed), or some other criteria.

In some embodiments, the task manager 112 (or another component of thedevice 110) marks a task item as complete once it has detected that thetask has actually been performed. In some embodiments, if a task itemhas an active payload, the task manager 112 determines that a task asbeen performed when an action associated with the active payload isperformed. For example, if a task item is a reminder to make a telephonecall to a person, the task manager 112 marks the task item as completewhen, after the task item is selected by a user, the device 110initiates a telephone to that person.

In some embodiments, task items are marked as complete based on one ormore rules that identify that a task has been or is likely to have beenperformed. In particular, the task manager 112 may be able to monitorvarious aspects of the device 110, such as network communication traffic(e.g., voice, internet protocol, etc.), API calls, and the like, todetermine whether tasks have been performed. In one example, if a taskitem is a reminder to send an email or text message, the task manager112 may detect whether, after selection of the task item, an email ortext message is actually sent by the device 110. Once the task manager112 detects that the email or text message has been sent, the task itemrelated to that action is marked as complete.

In another example, if a task item is a reminder to call someone, thetask manager 112 may detect whether a telephone call has been initiated.In some cases, this requires not only detecting whether a telephone callwas initiated, but also whether the telephone call was successful. Forexample, a user may initiate a telephone call, but the other party maynot pickup, or the call may go to voicemail. In these cases, it may beinappropriate to mark the task as complete. Accordingly, severaldifferent aspects of the device 110 may be monitored in order todetermine whether a telephone call task item has been successfullycompleted.

In some embodiments, the task manager 112 monitors a communicationinterface (e.g., communication interface 1618) of the device 110 todetect whether a telephone call has been initiated after the user hasselected the task item. In some embodiments, the task manager 112detects whether the telephone call results in a successful connection.In some embodiments, the task manager 112 monitors for voice input fromthe user that is characteristic of a telephone call. The task manager112 can use any of this information (and/or other information), alone orin combination, to determine whether the phone call was successfullycompleted.

In some embodiments, the task manager 112 can also determine that a taskwas not successfully completed in order to prevent the marking of a taskitem as complete. Specifically, a user may take certain actions on thedevice 110 that are inconsistent with the completion of a recentlyselected task item. For example, if a user selects a task item thatinitiates a phone call, but selects an “end call” button after a shortamount of time (or before the call is answered by a recipient), the taskmanager 112 can determine that the task item should not be marked ascomplete.

Indeed, any type of cancellation of a process that is necessary forcompletion of a task may be detected in order to determine that a taskhas not been completed. In some embodiments, device 1.10 includes one ormore control elements that are configured to cancel current operations,or that have an effect of cancelling certain operations. The controlelement may be an activation region of a touch screen, a power button, acontrol button, a switch, or the like. Button 634 (FIG. 6) illustratesan example of a control element that, when pressed, may cancel certainoperations and/or cause the device 110 to enter a different operatingstate.

In some embodiments, if a user activates (e.g., by touching, clicking,pressing, etc) a particular control element after a task item isselected, but before the task is completed, the task manager 112 willrecognize that the task item should not be marked as complete. Returningto the email example from above, if a task item is selected that causesthe display of a text input area, and the user selects a control elementthat has the effect of terminating the message before it is sent, thetask manager 112 will recognize that the task (i.e., sending an email toa particular recipient) was not completed.

Detecting events that indicate both task completion and tasknon-completion allows the task manager 112 to accurately andautomatically determine whether task items should be marked as complete.This helps to further automate the task list functions of the device110, as the user does not need to return to the task list and manuallymark task items as complete. Moreover, it prevents the task manager 112from marking task items as complete when they should not be.

Additionally or alternatively, task items that are consumed (i.e.,completed) are deleted from storage. For example, task manager 112deletes, from storage on device 110, any task items that have beenconsumed. The deletion of a task item may occur a certain period of time(e.g., 1 month) after the corresponding task has been completed to allowa user of device 110 to review recently-consumed task items. If a taskservice in cloud 130 manages task items that are stored in cloud 130,then that task service may delete consumed task items.

Attention is directed to FIG. 17, which illustrates a method 1700 thatincludes determining whether a task item should be marked as complete,according to an embodiment of the invention.

In some embodiments, the method 1700 includes detecting one or moreevents associated with the performing of the action (1716), anddetermining whether the one or more events satisfy one or moreconditions indicating that the action has been completed (1718). Thedifferent types of actions are described above with reference to step(1704).

In some embodiments, the action is sending an email, and the one or moreconditions are satisfied upon determining that an email has been sent bythe electronic device. In some embodiments, the action is initiating atelephone call, and the one or more conditions are satisfied uponreceiving a telephone call initiation request. In some embodiments, theaction is initiating a telephone call, and the one or more conditionsare satisfied upon detecting a telephone call connection event. Inembodiments where the action is initiating a telephone call, the one ormore conditions may be satisfied upon detecting a voice input that ischaracteristic of a telephone call.

In some embodiments, the method 1700 further includes detecting one ormore events associated with a failure to perform the action (1720), anddetermining whether the one or more events satisfy one or moreconditions indicating that the action has not been completed (1722).This helps to ensure that tasks are not marked as complete where theywere aborted after the task item was selected but before it was able tobe completed. In some embodiments, the one or more conditions aresatisfied upon receiving a user selection of a control element on theelectronic device. In some embodiments, the control element is selectedfrom the group consisting of an activation region of a touch screen, apower button, a control button, and a switch. In some embodiments wherethe action is initiating a telephone call, the one or more conditionsinclude not detecting a voice input within a predetermined time afterselection of the activation region.

Delayed Task Interpretation

In an embodiment, when a task item is created, only some details of thecorresponding task may be known and stored in association with the taskitem. Other details regarding the description, address (if any),trigger, and/or action may be determined later, whether automatically orvia a manual process.

For example, device 110 sends, to NLP service 132, voice data thatreflects a user command to “Call Sarah at 5.” NLP service 132 determinesthat 5 PM of the current day is a time trigger and causes task manager112 (or a task service in cloud 130) to create a task item with thattime trigger. However, an action item associated with the task item is“Call Sarah” without any indication of a phone number. NLP service 132has not yet determined who Sarah is and, thus, what phone number to useto call her. Instead, those details are determined later; for example,when the current time is 5 PM and the action is triggered or sometimebefore the trigger activates. At 5 PM, task manager 112 sends the actionitem “Call Sarah” (whether in text form or audio form) to NLP service132 or another service to identify information about a particular Sarah(if there are many) and to determine a phone number for Sarah. When aphone number for Sarah is determined, task manager 112 (or anotherprocess) causes a phone application on device 110 to initiate a callusing the phone number. In this example, the disambiguation of (a) theidentity of an individual and (b) a phone number for that individual isdelayed until after the task item is generated.

As another example, device 110 sends, to NLP service 132, voice datathat reflects a user command to “Check the weather in San Jose tomorrowmorning.” NLP service 132 determines that 7 AM of the next day is a timetrigger and causes task manager 112 (or a task service in cloud 130) tocreate a task item with that time trigger. However, an action itemassociated with the task item is “Check the weather in San Jose” withoutany indication of how to perform the action. NLP service 132 has not yetinterpreted that portion of the user command to determine how theweather in San Jose is to be checked. Instead, those details aredetermined later; for example, when the current time is 7 AM of the nextday and the action is triggered or sometime before the triggeractivates. At 7 AM of the next day, task manager 112 sends the actionitem “Check the weather in San Jose” (whether in text form or audioform) to NLP service 132 or another service to identify how the weatherin San Jose is to be checked. In response, NLP service 132 or anotherservice retrieves information about the weather in San Jose and providesthat information to device 110 to be displayed. In this example, thedetermination of how the action is to be performed is delayed untilafter the task item is generated.

Response to Alert

As alluded to previously, for a task item that is associated with anaction that is more than a mere notification, instead of performing theaction, a user of device 110 is first alerted of a task and the user isallowed to respond with an affirmative or negative response. Forexample, an action of a task item is to email Jane Smith about ProjectKnuckles. Task manager 112 causes, to be displayed on device 110, amessage that indicates that the user of device 110 is suppose to emailJane Smith. The user may press a physical or graphical button thatindicates an affirmative response. Alternatively, the user may speak thecommand, “Do it” or “Yes” indicating an affirmative response. Inresponse to the input (whether via a touch screen of device 110, akeyboard selection, or voice input), task manager 112 causes an emailapplication on device 110 to compose an email message addressed to JaneSmith with a subject line that refers to Project Knuckles.Alternatively, the user may decide to be reminded later of the task toemail Jane Smith. Thus, in response to the notification, the userprovides input (via device 110) that indicates that s/he would like toemail Jane Smith some time later, such as in one hour or the next day.Such input may be the user saying “Remind me later” or simply “later.”

In an embodiment, when the action is to respond to an act ofcommunication such as an email message, task manager 112 stores thecontext of the communication at the time of task creation and retrievesthe context at the time of performing the action. The context ofcommunication might be, in various embodiments, a Universal ResourceIdentifier or other reference to the context or a copy of the data ofthe context. For example, task manager 112 stores a reference to or copyof the email message that is to be replied to. When the action isperformed, the contents of the email message can be recreated just as ifthe user had performed a reply when initially reading it. Other examplesof context data that can be stored and retrieved in this manner includewithout limitation text messages, documents, web pages, voicemailmessages, photographs, audio recordings, and videos.

As another example, an action of a task item is to call George Burt. Inresponse to determining to trigger the action to call, task manager 112provides an indication that a reminder is available for a user of device110. The indication may be device 110 buzzing/shaking, generating anaudible noise, and/or displaying a notification message. Without holdingdevice 110, the user says aloud, “Read it.” In response to task manager112 (or another process) processing this input, device 110 plays anaudible version of the following statement: “Reminder . . . call GeorgeBurt.” The audible version may be based on a playback of the originalinput from the user or may reflect a computer-generated voice. If theuser decides to call George Burt, then the user may simply say, “Okay”or “Do it,” which causes a phone application on device 110 to callGeorge Burt. If the user decides not to call George Burt, then the usermay say, “Ignore” or “remind me later.”

IV. Organizing Task' Items Using Lists

According to an embodiment of the invention, a task item may beassociated with one or more lists. A list is a set of one or more taskitems that are associated with (or belong to) the same category. Listsare ways that a user of device 110 can view task items in an organizedway. The different lists allow the user to intelligently and intuitivelybrowse the tasks that s/he would like to perform (or have performed onhis/her behalf). FIGS. 6-14 depict views of various types of lists,according to an embodiment of the invention.

When a new task item is created, task manager 112 (or a service in cloud130) identifies one or more attributes associated with the new task itemand assigns the new task item to one or more lists. For example, if thenew task item includes the action “to call,” then task manager 112 (orother process) adds the new task item to a To Call list. Similarly, ifthe new task item includes a certain context and a particular location,then task manager 112 might identify the context and/or the particularlocation and add the new task item to a location list and/or a contextlist. Alternatively, a user might manually identify one or more of thelists, which are described in detail below, to which a new task item isto be added.

All Lists View

FIG. 5A depicts an All Lists view 500 that device 110 might display,according to an embodiment of the invention. All List view 500 does notcontain information about any specific task items. Instead, All Listsview 500 includes references to multiple lists maintained by taskmanager 112 (or a task service in cloud 130): a Today list 510, an AllTo Do list 520, a Nearby list 530, an In Car list 540, a To Call list550, a To Email list 560, a Groceries list 570, a To Buy list 580, and aCompleted list 590. As noted previously, a task item may be associatedwith (or belong to) multiple lists. For example, a task item whosedescription is to buy milk and whose time trigger is today may belong toToday list 510, All To Do list 520, Groceries list 570, and To Buy list580.

Lists may be characterized as one of three types: built-in or predefinedlist, smart list, or custom list. Today list 510, All To Do list 520,and Completed list 590 are examples of built-in or pre-defined lists.

Smart lists are based on different characteristics or attributes that atask item might have, such as an action (e.g., call, email, text,alert), a location, and/or a context in which the action is to beperformed. Examples of smart lists include By Action lists, By Locationlists, and By Context lists. In Car list 540, To Call list 550, and ToEmail list 560 are examples of By Action lists. Other examples of ByActions lists might include a To Text list, a To Lookup list, and a ToVisit list.

Examples of custom lists include lists that are based on categoriesidentified by NLP service 132 and lists that are created by a user.Groceries list 570 and To Buy list 580 are examples of custom lists.Another example of a custom list is a wine list (not shown) thatincludes a list of the user's favorite wines.

Returning to the lists depicted in FIG. 5A, task items that belong toToday list 510 are associated with a triggering criterion that indicatesa time during the current day that the corresponding task must or shouldbe performed. All task items belong to All To Do list 520. Task itemsthat belong to Nearby list 530 are associated with locations that areconsidered to be within a certain distance (e.g., 1 mile) from thecurrent location of device 110. Task items that belong to In Car list540 are associated with tasks that are to be performed in a car or whiletraveling. Task items that belong to To Call list 550 are associatedwith the action to call a person or entity. Task items that belong to ToEmail list 560 are associated with the action to email a person orentity. Task items that belong to Groceries list 570 are associated withgrocery items (e.g., milk, eggs, fruit) to purchase. Task items thatbelong to To Buy list 580 are associated with items to purchase, such asclothing, books, songs, or groceries. Task items that belong toCompleted list 590 are considered completed, which may indicate that thecorresponding tasks have been performed or at least that an action(e.g., an alert or notification) associated with each task item has beenperformed.

All Lists view 500 also includes a “+” image that when selected, allowsa user of device 110 to create another custom list so that currentand/or future task items can be added thereto.

FIG. 5B depicts some of the lists depicted in FIG. 5A, but with a searchfield 502 to allow a user of device 110 to search for a specific taskitem. A task item may be searched for based on, for example, the taskitem's associated creation date, completion date (if known), completionstatus, context trigger (if any), location (if any), and/or action type(e.g., notify only, call, email, or buy).

Today List

FIG. 6 depicts a view 600 of a Today list that device 110 displays, forexample, in response to user selection of Today list 510. View 600includes a list of tasks that are divided into two sections: a section610 for task items that are associated with a specific time and asection 620 for task items that are not associated with a specific time.Each of the task items in section 610 is associated with a travel timereminder. The third task item in section 610 and the second throughfourth task items in section 620 are associated with actions that aremore than mere reminders or alerts.

For example, the third task item in section 610 is to “pick up Chloe” at5:00 PM. The icon to the right of that description is an image of acompass, indicating that the action associated with this task item is togenerate travel directions to help guide the user of device 110 to theintended destination, which is Pinewood School in this example.

As another example, the second task item in section 620 is to “call JohnAppleseed.” The icon to the right of that description is an image of aphone, indicating that the action associated with this task item is tocall John Appleseed. The image adjacent to the phone image is of a car,indicating that the user of device 110 is to call John Appleseed whenthe user is in a car or while the user is traveling.

As another example, the last task item in section 620 is to “reply toPablo Marc.” The icon to the right of that description is an image of anenvelope, indicating that the action associated with this task item isto send an email to Pablo Marc. View 600 also indicates that this taskitem is overdue, or rather, that the originally-scheduled time to emailPablo Marc has passed.

Single Task Item View

FIG. 7 depicts a view 700 that device 110 displays and that includesdetails about a particular task item. View 700 may have been generatedbased on a user selection of the second task item in section 620 in view600 of FIG. 6. The displayed task item contains four data items: adescription item 710, an action item 720, a reminder item 730, and alist assignment item 740.

Description item 710 contains a high-level description of the task(“Call John Appleseed”) and includes details about the subject matter(“Discuss the almond deal”). Selection of description item 710 may allowa user of device 110 to edit the description.

Action item 720 contains a description of the action (“Call”) andincludes which phone (“mobile”) of John Appleseed to use. Selection ofaction item 720 may allow the user of device 110 to view the phonenumber associated with John Appleseed and/or provide other contactoptions, such as another phone number associated with John Appleseed, anemail address of John Appleseed, etc. Furthermore, selection of thephone icon in action item 720 may cause task manager 112 to initiate acall phone to John Appleseed right then instead of waiting for the oneor more triggering criteria associated with the task item to besatisfied.

Reminder item 730 indicates the type of trigger (“when in car”) that,when detected, will cause the action to be performed, or at least analert about the task. Selection of reminder item 730 may allow a user tochange the type of reminder.

List assignment item 740 indicates the list to which the task itembelongs, which is the “Nut to Crack Project” list in this example. Thislist is an example of a customized list. Selection of list assignmentitem 740 may cause device 110 to display multiple task items that belongto the “Nut to Crack Project” list.

All To Do List

FIG. 8 depicts a view 800 of an All To Do list that device 110 displaysand that includes information about multiple task items. In thisexample, the multiple task items are ordered by date. View 800 may havebeen generated based on a user selection of All To Do list 820 in view800 of FIG. 8A. View 800 is divided into two sections: section 810 thatcontains task items (or references thereto) to be completed on one dayand section 820 that contains task items to be completed on thefollowing day.

Some of the task items referenced in view 800 have been completed. Suchcompleted task items are shown with a lighter gray image to the left ofthe corresponding description. Task items that have been completed maybe distinguished from not-yet-completed task items by other techniques,such as check marks.

In the example depicted in FIG. 8, the task items are organized by thedate on which the corresponding tasks should be performed (or “duedate”). However, the task items referenced in view 800 may be organizedby the date on which a user of device 110 is to be alerted or remindedof the corresponding tasks (“alert date”), the date on which the taskitems were created (“created date”), the date on which the task itemswere modified (“modified date”), or the date on which the correspondingtasks were performed (“completed date”).

Nearby List

FIG. 9 depicts a view 900 of a “Nearby” list that device 110 displays.View 900 may have been generated based on a user selection of Nearbylist 830 in view 800 of FIG. 8A. View 900 contains information aboutmultiple locations that are ordered based on distance from device 110'scurrent location. The location indicated at the top of the list (“Home”)is closest to the current location of device 110 while the locationindicated at the bottom of the list (“Pinewood School”) is furthest fromthe current location of device 110.

Each location indicated in view 900 is associated with a differentlocation list. Each location list may be associated with one or moretask items. For example, the “Home” location may be associated with fourtask items (which may be displayed on user selected of the “Home”location) while the “Atherton Dry Cleaning” location may be associatedwith just one task item.

Because the locations indicated in view 900 are ordered based ondistance from the current location of device 110, when the currentlocation of device 110 changes, the location indicators may bere-ordered, some may be removed from view 900, and others not currentlydisplayed in view 900 may appear in view 900. For example, if device 110is currently located in a store that is next to the Whole Foods storeidentified by the second location indicated in view 900, then, if device110 displays view 900, that Whole Foods location indicator will be atthe top of the list.

As indicated above, view 900 includes a “Home” location and a “Work”location. The association of a location labeled “Home” (or “Work”) witha particular address may be made in numerous ways. For example, manymobile devices store profile information about a user of the mobiledevice. This information is referred to as a “me card.” A me cardtypically stores a user's home address and the user's work address.Thus, task manager 112 (or another process) analyzes the me card that isstored on device 110 to determine a home address and a work address (ifany) of the user.

In an embodiment, a radius is associated with a particular location andany task items that are associated with a location that is within thedistance indicated by the radius is considered to be associated with theparticular location. For example, a radius associated with a home of auser of device 110 is 2 miles. If a task item is associated with a parkand the park is within 2 miles from the home, then the task item isassociated with a “home” list, along with other task items that areassociated with the home.

Location List View

As noted previously, a location list is an example of a smart list. Inan embodiment, any task item that is associated with a location (e.g.,as part of the one or more triggering criteria) is automaticallyassociated with a location list that is associated with the samelocation as the location of the task item. Task manager 112 (or a taskservice in cloud 130) may maintain multiple location lists.

FIG. 10A depicts a Location List view 1000 that device 110 displays.Location list view 1000 may have been generated based on a userselection the “Home” location indicator in Nearby view 900 of FIG. 9.Location list view 1000 contains six task items. The bell image adjacentto each of the first four task items indicates that a reminder (oralert) for those task items will be generated when device 110 is at ornear the user's home or at least sometime on a specified date. Areminder or alert will not be generated for the last two task items.

Location List view 1000 also includes a map icon 1002 which, whenselected, causes task manager 112 to communicate with a map applicationthat generates a map of the location associated with the map icon. Inthis example, a map of the user's home would be generated.

FIG. 10B depicts a Location List view 1050 that device 110 displays.Location List view 1050 may have been generated based on a userselection the “Whole Foods” location indicator in Nearby view 900 ofFIG. 9. Location List view 1050 contains six data items, each of whichmay or may not be a task item. Instead, each data item in Location Listview 1050 simply identifies a grocery item to purchase at a Whole Foodsgrocery store. None of the grocery items are associated with a reminder(although they could be) or a completion date (although they could be).

The grocery items identified in Location List view 1050 was associatedwith the Whole Foods grocery list in response to input from a user ofdevice 110. For example, a user spoke the following command: “Add almondmilk to my grocery list” or “Remember to pick up almond milk at WholeFoods near my house.” Device 110 transmits voice data that reflects thiscommand to NLP service 132. NLP service 132 determines, based on thevoice data, that the user intends to purchase almond milk. NLP service132 may cause task manager 112 to (a) create a task item for the task ofpurchasing almond milk and add the task item to the Whole Foods list or(b) simply add “almond milk” to the Whole Foods list.

Location List view 1050 also includes a map icon 1052 which, whenselected, causes task manager 112 to communicate with a map applicationthat generates a map of the location associated with the map icon. Inthis example, a map of the Whole Foods store identified by the displayedaddress would be generated.

Smart Lists

As noted previously, By Location lists, By Action lists, and By Contextlists are examples of smart lists. FIG. 11A depicts a view 1100 of a ByContext list; specifically, an In Car list. FIG. 11B and FIG. 11D depictviews of different By Action lists; specifically, a To Call list and aTo Email list.

View 1100 contains task items that are associated with tasks that are tobe performed in a specific context, i.e., the “In Car” context. The taskitems in the In Car list may be associated with different actions, suchas calling and getting directions.

In contrast, view 1110, depicted in FIG. 11B, contains task items thatare associated with the same action, which, in this example, is to calla person or entity. The first three task items in view 1110 have a phoneicon, indicating that a phone number for the person indicated in thecorresponding task is known to task manager 112. However, the last taskitem in view 1110 is not associated with a phone icon, indicating that aphone number for “Bob” is not positively known to task manager 112,probably because many contacts in the user's contact list may have thename of Bob. Selection of the “call Bob” task item in view 1110 causesdevice 110 to display a view 1120 depicted in FIG. 11C.

View 1120 indicates two data items that are contained in (or associatedwith) the “call Bob” task item: a description item and an action item.The action item indicates that multiple contacts are known as “Bob.” Asa result, the action item includes a call button that is disabled,whereas the call buttons associated with the other task items in view1110 are not disabled. Selection of the action item may initiate aprocess for disambiguating the identity of “Bob.” For example, selectionof the action item may cause task manager 112 to display a list ofnames, each of which have the name of Bob or Robert. In this way, thedisambiguation of an identity or of a phone number may occur much laterthan the creation of the corresponding task item.

View 1130, depicted in FIG. 11D, includes six task items, each of whichincludes an action to email. The active payload arguments of a To Emailtask item include a “To” or email address and, optionally, a subject forthe subject line of the email.

In an embodiment, an “email” task item is created from an emailapplication that is separate from task manager 112. The emailapplication may invoke an API call of task manager 112 to create a taskitem whose action is to email, where the action includes an activepayload that includes an email address and a subject.

Custom Lists

As noted previously, custom lists are one of the three main types oflists, including built-in lists and smart lists. Examples of customlists indicated above include Grocery list 570 and To Buy list 580(referenced in FIG. 5A). FIG. 12 depicts a view 1200 that might begenerated in response to user selection of Grocery list 570. View 1200includes six data items, each referring a different grocery item topurchase. Each of these data items may be task items that only have adescription. The data items may have been associated with the grocerylist based on input from NLP service 132. For example, NLP servicereceives, from device 110, voice data that reflects the user command to“pick up fresh bread from the store.” NLP service 132 determines thatthe user of device 110 intends to purchase fresh bread from a grocerystore and associates “fresh bread” with a grocery category. In response,NLP service 132 sends, to task manager 112, a create task item commandto create a task item that includes the description “fresh bread” andthat is associated with the grocery category. In response, task manager112 creates a task item and associates the task item with a grocery listthat task manager 112 maintains.

FIG. 13 depicts a view 1300 of another type of custom list: auser-defined list. This user-defined list is entitled, “Nut to CrackProject,” and contains three task items, the first of which isassociated with an action (i.e., call) and a context trigger (e.g. “incar” or “while driving”). A user of device 110 may “manually” associatea task item with a user-defined list. For example, after task manager112 creates a task item, the user selects the task item and, via one ormore selectable (e.g., menu) options displayed on device 110, selects aparticular user-defined list, which causes task manager 112 to associatethe task item with the particular user-defined list.

Alternatively, NLP service 132 may determine, based on input data(whether voice or text) received from device 110, a specific list toassociate with a task item. For example, voice data may reflect a usercommand to “I need to write a proposal for the Nut to Crack Project.”NLP service 132 determines that “write a proposal” is the task and that“Nut to Crack Project” is the name of a list, which task manager 112 mayor may not have yet created. NLP service 132 then sends, to task manager112, the description (“write proposal”) and the name of a possible listto which the to-be-created task item may be added (“Nut to CrackProject”). Task manager 112 determines whether there is a list that hasthe same or similar name as “Nut to Crack Project.” If so, then taskmanager 112 creates a new task item and associates the task item withthat list. If not, then task manager 112 creates a new list with thatname, creates a new task item, and associates that task item with thenew list.

Lists And Notes

As noted previously, a list may contain items that are not tasks. Such“non-task” are referred to as “notes” that consist only of adescription. FIG. 14 depicts a view 1400 of a Favorite Wines list, whichcontains six notes, each referring to a different wine.

Also as noted previously, NLP service 132 may be configured to recognizelist names so that task manager 112 can easily assign tasks and notes tothe appropriate list(s).

Calendar Events

In an embodiment, calendar events created in the context of a calendarapplication are used to create task items that are managed by taskmanager 112. The calendar application may be part of task manager 112 ormay be separately executing applications. For example, the calendarapplication might be configured to send newly-created calendar events totask manager 112, e.g., via one or more API calls that cause taskmanager 112 to create a task item based on the details of a calendarevent, such as a description, a date, a location (if any), a duration(if any), and a reminder (if any). Alternatively, task manager 112 mightprovide a calendar service that allows a user to view a calendar andcreate events that are associated with a specific date and time or setof dates. Upon creation of events, task manager 112 also creates taskitems for the events.

FIG. 15 depicts a view 1500 of a task item that was generated based on acalendar event. The task item includes four data items: a description(“lunch with Elizabeth Reid”), a begin time (“12:00 PM Today”), aduration (“1 hour”), and a reminder (“10 minutes before”). Selection ofany of the four data items may allow a user of device 110 to edit thecorresponding data items. In an embodiment, if a change is made to atask item that was generated based on a calendar event, then that changeis “pushed” to the calendar event that is managed by a calendarapplication.

In either scenario, if a calendar event that is created and maintainedby the calendar service is associated with a location, then a task itemthat is generated based on the calendar event might also be associatedwith the location. In that case, task manager 112 might automaticallyassociate the task item with a location list, such as the location listin view 1000 of FIG. 10A.

Combinations

While the foregoing description includes four main approaches(generating task items, organizing task items, triggering notifications,and consuming task items), each of these approaches may be implementedindividually or may be used together, as noted in many of the examples.For example, natural language processing may be used to generate a taskitem, but none of the approaches described herein for processing thetask item (i.e., organizing the task item, triggering a notification,and consuming the task item) are used. As another example, naturallanguage processing may be used to generate a task item and an approachfor organizing the task item as described herein may be used, but noneof the approaches for triggering a notification or consuming the taskitem described herein are used. As another example, none of theapproaches for generating and organizing task items and triggering anotification is used, but the approach for consuming the task item asdescribed herein is used.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 16 is a block diagram that illustrates a computersystem 1600 upon which an embodiment of the invention may beimplemented. Computer system 1600 includes a bus 1602 or othercommunication mechanism for communicating information, and a hardwareprocessor 1604 coupled with bus 1602 for processing information.Hardware processor 1604 may be, for example, a general purposemicroprocessor.

Computer system 1600 also includes a main memory 1606, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1602for storing information and instructions to be executed by processor1604. Main memory 1606 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1604. Such instructions, when stored innon-transitory storage media accessible to processor 1604, rendercomputer system 1600 into a special-purpose machine that is customizedto perform the operations specified in the instructions.

Computer system 1600 further includes a read only memory (ROM) 1608 orother static storage device coupled to bus 1602 for storing staticinformation and instructions for processor 1604. A storage device 1610,such as a magnetic disk or optical disk, is provided and coupled to bus1602 for storing information and instructions.

Computer system 1600 may be coupled via bus 1602 to a display 1612, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1614, including alphanumeric and other keys, iscoupled to bus 1602 for communicating information and command selectionsto processor 1604. Another type of user input device is cursor control1616, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1604 and for controlling cursor movement on display 1612. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1600 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1600 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1600 in response to processor 1604 executing one or moresequences of one or more instructions contained in main memory 1606.Such instructions may be read into main memory 1606 from another storagemedium, such as storage device 1610. Execution of the sequences ofinstructions contained in main memory 1606 causes processor 1604 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 1610.Volatile media includes dynamic memory, such as main memory 1606. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1602. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1604 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1600 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1602. Bus 1602 carries the data tomain memory 1606, from which processor 1604 retrieves and executes theinstructions. The instructions received by main memory 1606 mayoptionally be stored on storage device 1610 either before or afterexecution by processor 1604.

Computer system 1600 also includes a communication interface 1618coupled to bus 1602. Communication interface 1618 provides a two-waydata communication coupling to a network link 1620 that is connected toa local network 1622. For example, communication interface 1618 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1618 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1618 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1620 typically provides data communication through one ormore networks to other data devices. For example, network link 1620 mayprovide a connection through local network 1622 to a host computer 1624or to data equipment operated by an Internet Service Provider (ISP)1626. ISP 1626 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1628. Local network 1622 and Internet 1628 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1620 and through communication interface 1618, which carrythe digital data to and from computer system 1600, are example forms oftransmission media.

Computer system 1600 can send messages and receive data, includingprogram code, through the network(s), network link 1620 andcommunication interface 1618. In the Internet example, a server 1630might transmit a requested code for an application program throughInternet 1628, ISP 1626, local network 1622 and communication interface1618.

The received code may be executed by processor 1604 as it is received,and/or stored in storage device 1610, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method for triggering an action associated witha task, comprising: at an electronic device having one or moreprocessors and memory storing one or more programs for execution by theone or more processors: receiving a task item associated with atriggering criterion, wherein the triggering criterion requires anoccurrence of any of a plurality of communication events, wherein theplurality of communication events includes a telephone call; detectingthe occurrence of a first communication event of the plurality ofcommunication events; determining that the triggering criterion issatisfied; and upon determining that the triggering criterion issatisfied, causing a notification associated with the task item to bepresented to a user of the electronic device.
 2. The method of claim 1,wherein the first communication event is a telephone call.
 3. The methodof claim 2, wherein at least two of the plurality of communicationevents are of different communication types.
 4. The method of claim 3,wherein the different communication types include a telephone call andan email.
 5. The method of claim 3, wherein the different communicationtypes include a telephone call and a text message.
 6. The method ofclaim 3, wherein the different communication types include a telephonecall and a video chat.
 7. The method of claim 1, wherein the firstcommunication event includes content, and wherein the triggeringcriterion requires that the content relate to a predefined subject, themethod further comprising: determining that the triggering criterion issatisfied based at least in part on a determination that the contentrelates to the predefined subject.
 8. The method of claim 7, wherein thecontent is text from an email or a text message.
 9. The method of claim7, wherein the first communication event is a telephone call, the methodfurther comprising: obtaining a text string corresponding to one or moreutterances spoken during the telephone call by at least one participantin the telephone call; wherein the determination that the contentrelates to the predefined subject includes determining that one or morewords in the text string relate to the predefined subject.
 10. Themethod of claim 1, wherein the triggering criterion further requiresthat the communication event be a communication with a specified person.11. The method of claim 1, wherein the first communication event is atext message, and causing the notification to be presented comprisescausing an affordance relating to the first task item to be displayed onthe electronic device near a representation of the text message.
 12. Themethod of claim 11, wherein the affordance is a button that, whenselected by the user, causes an action associated with the task item tobe performed.
 13. The method of claim 1, wherein the first communicationevent is an email, and causing the notification to be presentedcomprises causing an affordance relating to the first task item to bedisplayed on the electronic device near a representation of the email.14. The method of claim 13, wherein the affordance is a button that,when selected by the user, causes an action associated with the taskitem to be performed.
 15. The method of claim 1, wherein the firstcommunication event is a telephone call, and causing the notification tobe presented comprises causing an audio output relating to the firsttask item to be produced by the electronic device.
 16. The method ofclaim 15, wherein the electronic device is a telephone, and the audiooutput is produced while the telephone is ringing.
 17. The method ofclaim 15, wherein the audio output is produced during a telephoneconversation between a user of the electronic device and a participantof the telephone call.
 18. The method of claim 17, wherein the audiooutput is produced at a lower volume than a voice of the participant ofthe telephone call.
 19. A non-transitory computer readable storagemedium storing one or more programs configured for execution by anelectronic device, the one or more programs comprising instructions for:receiving a task item associated with a triggering criterion, whereinthe triggering criterion requires an occurrence of any of a plurality ofcommunication events, wherein the plurality of communication eventsincludes a telephone call; detecting the occurrence of a firstcommunication event of the plurality of communication events;determining that the triggering criterion is satisfied; and upondetermining that the triggering criterion is satisfied, causing anotification associated with the task item to be presented to a user ofthe electronic device.
 20. A method for triggering an action associatedwith a task, comprising: at an electronic device having one or moreprocessors and memory storing one or more programs for execution by theone or more processors: providing a task item associated with atriggering criterion, wherein the triggering criterion requires anoccurrence of any of a plurality of communication events, wherein atleast two of the plurality of communication events are of differentcommunication types; detecting the occurrence of a first communicationevent of the plurality of communication events; determining that thetriggering criterion is satisfied; and upon determining that thetriggering criterion is satisfied, causing a notification associatedwith the task item to be presented to a user of the electronic device.21. The method of claim 20, wherein the triggering criterion requiresthat the first communication event be associated with a specifiedperson.
 22. The method of claim 21, wherein the first communicationevent is associated with the specified person when the specified personinitiates the first communication event or receives the firstcommunication event from the user.
 23. A method for triggering an actionassociated with a task, comprising: at an electronic device having oneor more processors and memory storing one or more programs for executionby the one or more processors: providing a task item associated with atriggering criterion, wherein the triggering criterion requires anoccurrence of a telephone call at an electronic device; detecting theoccurrence of the telephone call; and in response to detecting theoccurrence of the telephone call, causing a notification associated withthe task item to be presented to a user of the electronic device,wherein the notification is separate from a telephone call notification.